Benchmark operator
VideoGameBench
Tests whether vision-language models can finish classic games from raw frames alone, including Pokémon Crystal.
vgbench.com
Models: GPT-4o (2024-08-06), Gemini 2.5 Pro (preview 03-25)
Games: Kirby's Dream Land, Pokémon Crystal
Achievement history
VideoGameBench
| Model | Game | Achievement | Run config | Status | Date |
|---|
| GPT-4o (2024-08-06) |
Pokémon Crystal |
Memorable fail 0.9% of Pokémon Crystal completed on VideoGameBench (test set, real-time) Picked Cyndaquil as its starter, then lost track of what it was supposed to do. Last verified: Oct 18, 2026 · Last checked: Oct 18, 2026 How was this run measured?
- Model
- GPT-4o (2024-08-06)
- Game
- Pokémon Crystal
- Project
- VideoGameBench
- Harness
- VG-Agent (VideoGameBench reference agent)
- Inputs
- raw game frames + high-level objective and controls
- Tools
- none
- Assistance
- none
- Budget
- real-time play (game does not pause)
- Verification
- Verified Official leaderboard published by the benchmark authors.
- Evidence
- VideoGameBench leaderboard (official) (operator)
VideoGameBench paper (operator)
- Dates
- evidence dated May 23, 2025
- Recorded
- published May 23, 2025 · discovered by AI vs Games Oct 18, 2026
- Changes
- Historical record. Nothing to update unless the source changes.
Permanent link · Submit a correction · JSON |
VG-Agent (VideoGameBench reference agent) raw game frames + high-level objective and controls |
Verified |
May 23, 2025 |
| Gemini 2.5 Pro (preview 03-25) |
Kirby's Dream Land |
Memorable fail 4.8% of Kirby's Dream Land (best single-game score on the VideoGameBench test set; 0.48% overall) Last verified: Oct 18, 2026 · Last checked: Oct 18, 2026 How was this run measured?
- Model
- Gemini 2.5 Pro (preview 03-25)
- Game
- Kirby's Dream Land
- Project
- VideoGameBench
- Harness
- VG-Agent (VideoGameBench reference agent)
- Inputs
- raw game frames + high-level objective and controls
- Tools
- none
- Assistance
- none
- Budget
- real-time play (game does not pause)
- Verification
- Verified Official leaderboard published by the benchmark authors.
- Evidence
- VideoGameBench leaderboard (official) (operator)
VideoGameBench paper (operator)
- Dates
- evidence dated May 23, 2025
- Recorded
- published May 23, 2025 · discovered by AI vs Games Oct 18, 2026
- Changes
- Historical record. Nothing to update unless the source changes.
Permanent link · Submit a correction · JSON |
VG-Agent (VideoGameBench reference agent) raw game frames + high-level objective and controls |
Verified |
May 23, 2025 |
Are you behind this project? Get in touch to correct anything or to add evidence.