Benchmark operator

PokéAgent Challenge

A Pokémon benchmark with a battling track and a speedrunning track. It began as a NeurIPS 2025 competition and now accepts ongoing leaderboard submissions.

pokeagentchallenge.com · Paper

Models: Deepest agent (model not disclosed), Hamburg PokéRunners agent (model not disclosed), Heatz agent (model not disclosed), anthonys agent (model not disclosed)
Games: Pokémon Emerald

Achievement history

PokéAgent Challenge
Model	Game	Achievement	Run config	Status	Date
Heatz agent (model not disclosed) Heatz	Pokémon Emerald	1st: reached the first gym (Roxanne) in 40:13 (1,608 actions) Nearly twice as fast as 2nd place. Last verified: Oct 18, 2026 · Last checked: Oct 18, 2026 How was this run measured? Model Heatz agent (model not disclosed) · agent: Heatz Game Pokémon Emerald Project PokéAgent Challenge (NeurIPS 2025, speedrun track) Harness PokéAgent Challenge Emerald environment (real-time; game keeps running while the agent thinks) Inputs per team (documented in the paper's appendix) Tools Scripted Policy Distillation: an LLM writes scripted policies, distilled into a neural net and refined with RL Assistance none allowed during evaluation runs Budget scored on wall-clock time and action count Verification Verified Official results published by the challenge organizers. Evidence PokéAgent Challenge paper (organizers), Sec. 5.2 & App. C (operator) PokéAgent Challenge: Speedrunning track (operator) Unknown underlying model(s) used by the team (not in our source) Dates evidence dated Mar 16, 2026 Recorded published Mar 16, 2026 · discovered by AI vs Games Oct 18, 2026 Changes Historical record. Nothing to update unless the source changes. Permanent link · Submit a correction · Share card (PNG) · JSON	PokéAgent Challenge Emerald environment (real-time; game keeps running while the agent thinks) per team (documented in the paper's appendix)	Verified source	Mar 16, 2026
Hamburg PokéRunners agent (model not disclosed) Hamburg PokéRunners	Pokémon Emerald	2nd: reached the first gym in 1:14:43 Last verified: Oct 18, 2026 · Last checked: Oct 18, 2026 How was this run measured? Model Hamburg PokéRunners agent (model not disclosed) · agent: Hamburg PokéRunners Game Pokémon Emerald Project PokéAgent Challenge (NeurIPS 2025, speedrun track) Harness PokéAgent Challenge Emerald environment (real-time; game keeps running while the agent thinks) Inputs per team (documented in the paper's appendix) Tools Pure RL: recurrent PPO with milestone-based rewards Assistance none allowed during evaluation runs Budget scored on wall-clock time and action count Verification Verified Official results published by the challenge organizers. Evidence PokéAgent Challenge paper (organizers), Sec. 5.2 & App. C (operator) PokéAgent Challenge: Speedrunning track (operator) Unknown underlying model(s) used by the team (not in our source) Dates evidence dated Mar 16, 2026 Recorded published Mar 16, 2026 · discovered by AI vs Games Oct 18, 2026 Changes Historical record. Nothing to update unless the source changes. Permanent link · Submit a correction · JSON	PokéAgent Challenge Emerald environment (real-time; game keeps running while the agent thinks) per team (documented in the paper's appendix)	Verified source	Mar 16, 2026
anthonys agent (model not disclosed) anthonys	Pokémon Emerald	3rd: reached the first gym in 1:29:17 Last verified: Oct 18, 2026 · Last checked: Oct 18, 2026 How was this run measured? Model anthonys agent (model not disclosed) · agent: anthonys Game Pokémon Emerald Project PokéAgent Challenge (NeurIPS 2025, speedrun track) Harness PokéAgent Challenge Emerald environment (real-time; game keeps running while the agent thinks) Inputs per team (documented in the paper's appendix) Tools Tool-calling VLM policy (LLM harness) Assistance none allowed during evaluation runs Budget scored on wall-clock time and action count Verification Verified Official results published by the challenge organizers. Evidence PokéAgent Challenge paper (organizers), Sec. 5.2 & App. C (operator) PokéAgent Challenge: Speedrunning track (operator) Unknown underlying model(s) used by the team (not in our source) Dates evidence dated Mar 16, 2026 Recorded published Mar 16, 2026 · discovered by AI vs Games Oct 18, 2026 Changes Historical record. Nothing to update unless the source changes. Permanent link · Submit a correction · JSON	PokéAgent Challenge Emerald environment (real-time; game keeps running while the agent thinks) per team (documented in the paper's appendix)	Verified source	Mar 16, 2026
Deepest agent (model not disclosed) Deepest	Pokémon Emerald	Judge's Choice: fewest actions to the first gym (649), 5th by time Slowest clock, fewest button presses: a good example of why we never merge metrics. Last verified: Oct 18, 2026 · Last checked: Oct 18, 2026 How was this run measured? Model Deepest agent (model not disclosed) · agent: Deepest Game Pokémon Emerald Project PokéAgent Challenge (NeurIPS 2025, speedrun track) Harness PokéAgent Challenge Emerald environment (real-time; game keeps running while the agent thinks) Inputs per team (documented in the paper's appendix) Tools LLM harness Assistance none allowed during evaluation runs Budget scored on wall-clock time and action count Verification Verified Official results published by the challenge organizers. Evidence PokéAgent Challenge paper (organizers), Sec. 5.2 & App. C (operator) PokéAgent Challenge: Speedrunning track (operator) Unknown underlying model(s) used by the team (not in our source) Dates evidence dated Mar 16, 2026 Recorded published Mar 16, 2026 · discovered by AI vs Games Oct 18, 2026 Changes Historical record. Nothing to update unless the source changes. Permanent link · Submit a correction · JSON	PokéAgent Challenge Emerald environment (real-time; game keeps running while the agent thinks) per team (documented in the paper's appendix)	Verified source	Mar 16, 2026

Are you behind this project? Get in touch to correct anything or to add evidence.