A new benchmark tests AI planning in interactive environments using five thousand hours of gameplay and one million calls

A new research paper reveals the development of the Game Horizon Suite, a unified data and evaluation platform intended to measure AI models' abilities to play video games across multiple time horizons. This step aims to address structural shortcomings in current evaluation benchmarks, which have remained limited to narrow-scope games, lacking directed language instructions, or relying on interactive play sessions with wide statistical variance that hampers disciplined scientific replication.

The system starts from the fact that modern video games have become a standard, measurable experimental environment for testing generative and agentic models, as they simultaneously combine visual scene understanding, decomposition of complex instructions into sub-steps, strategic goal planning, and precise motor control across time trajectories of varying lengths.It establishes a three-part integrated architecture that merges automated annotation tools, databases, and comparative testing platforms.to provide a standardized measurement track that exceeds usual partial evaluations.

The system’s architecture relies on three interlinked tracks: the first, Game Horizon Annotator, is an automated, scalable workflow for generating multi-horizon instruction annotations. The second, Game Horizon Data, serves as the first large-scale database dedicated to mass-production games, containing five thousand hours of recordings from twenty-one popular titles, collected by a hundred professional human players, with precise, synchronized alignment of video clips, player motor actions, and linguistic commands. The third, Game Horizon Bench, offers dual evaluation: an offline track that employs thousands of benchmark questions categorized into three primary tasks and alternative diagnostic tests, and an online track that assesses the correspondence of theoretical results to real-world performance, pinpointing exactly where models fail during long-duration gameplay.

The test results of forty-seven models across more than a million calls reveal a fundamental variance in response capabilities and continuous planning.The evaluation showed a clear gradient in task difficulty and essential differences between model software families, with researchers committing to make the full database and evaluation tools publicly available to the scientific community to advance future research.

This work marks a decisive shift in how digital agents are tested, moving the focus from static conversational and textual inference tests to actionable capabilities and control within complex dynamic environments. For developers and AI engineering teams in the region, especially in Saudi Arabia and the United Arab Emirates as regional investments in digital gaming and autonomous agents grow, the availability of standardized, open evaluation environments saves startups and research labs the high costs previously spent building isolated internal assessment tools. Moreover, this approach mandates training local technical talent in fault isolation mechanisms and developing models capable of planning across extended time horizons rather than relying on momentary responses, which defines the true gap between a prototype model and a reliable, commercially viable digital agent in real-world work settings.