Programming agent tests extend to three hours, a metric tracking models’ staged progress

Tech companies evaluate AI agents with a decisive binary test: either the programming task succeeds completely or the algorithm records a total failure. This exclusionary logic ignores the nature of complex work that spans continuous hours within terminal command environments. A new research paper authored by 17 researchers on the "Orcive" platform reveals that measuring only final achievement wastes a substantial portion of programming effort, introducing an experimental framework that tests models’ resilience and cumulative progress in three-hour-long work streams.

The updated framework, named "Long-Horizon-Terminal-Bench 2.0", expands the previous benchmark from 46 original tasks to 77 tasks spread across 13 scientific and engineering domains and eight task types, after fixing evaluation infrastructure flaws and adding 31 tasks. The researchers adopted a methodological path to rebuild tasks using functional engineering and programming tools, generating multiple scenarios, six of which are revealed as behavioral evidence for the model, while three scenarios are withheld for the final test.The engineering of graded progress ladders prevented models from memorizing declared outputs and focused on rewarding partial achievement at each step.

The most notable feature of the new benchmark is extending the temporal testing horizon from 90 minutes to a full three hours, with automatic checkpoints every 30 minutes that precisely track the model’s trajectory and whether it maintains its direction or deviates midway. Across 1,145 test runs covering 15 leading advanced models, the "GPT-6-astra" model topped the results with an average cumulative reward of 0.69 and achieved a completion ratio above 0.95 on 37 tasks, followed closely by "GPT-5.6-sol" with a rate of 0.67 and reaching the same threshold on 32 tasks.

In the study’s details we examined the digital gap recorded by the test results, finding it provides practical evidence of the shortcomings of traditional evaluations; across thirty tasks dedicated to reconstruction, the average score of binary verification tools measuring the final outcome did not exceed 0.221, whereas the average score of the graded progress ladder rose to 0.653.Models successfully traverse two-thirds of the engineering path before stumbling on the final details, making a judgment of absolute failure misleading for development teams.

This shift in measurement philosophy imposes a direct change on engineering teams and digital transformation departments in the Gulf, Egypt, and the Levant, where banks and logistics firms race to employ independent agents for automating complex systems. Continuing to rely on binary success criteria either rushes the dismissal of promising software solutions that handle most of the work skillfully, or risks deploying agents without sufficient staged oversight. The practical decision today is to build audit platforms and software inspection stations every half hour within the operating environment, enabling your team to detect deviation early and reduce wasted computing and inference costs in failed attempts that only reveal errors at the finish line.

The experiment’s results show that the maturity of AI agents in long tasks will not be measured by lucky hits that affect the final output, but by the system’s ability to continue systematic construction step by step. As benchmarks shift from merely tracking the final answer to accompanying the agent along a multi-hour timeline, the rules of the game change: the true competitive advantage now lies in engineering staged auditing, not in awaiting a blind result.