Spire Bench tests AI agents, showing early parity with experts and collapse in extended decisions

Business automation experiments reveal that language models can perform short tasks with striking accuracy, but their ability to engage in strategic thinking over extended trajectories remains a unresolved scientific question. Most agent tests rely on brief, fragmented tasks, making it difficult to separate genuine performance and determine whether it is driven by memorized data, authentic logical inference, or statistical chance. In this context, researchers Scott Loew and Sandra Keminick presented, via a study on the “Archive” platform, a new testing environment called “Spire Bench”, which gives models the full commercial game “Slay the Spire”, where early decisions determine win or loss through a long, complex strategic sequence.

The researchers subjected twenty large language models to the game's highest difficulty level, and the results highlighted a fundamental gap between momentary tactics and long-term vision. State-of-the-art models outperformed traditional baselines and matched the performance of an average human player, yet they remained far behind expert level. This gap widened noticeably as the trajectory length increased; the strongest models matched expert win rates in the game's first stage, then declined sharply in later stages.

What separates strong models from weak ones is efficiency in managing individual battles, while all of them share a strategic inability to assess risk over the full trajectory.The examination showed that models tend to safeguard their immediate health points rather than make calculated sacrifices of some of those points to strengthen themselves and acquire items that grant immunity in later stages. This reluctance to expend a transient resource to build cumulative capability explains agents' failures in environments that require temporary risk-taking to reach a broader goal.

To verify the nature of this performance, the researchers performed a comprehensive, self-consistent renaming of every term and rule within the game.The results demonstrated that a tangible portion of the models' superiority relied on retrieving previously stored knowledge from training data, rather than on live inference that interacts with newly introduced rules.Moreover, the experiment tested the models' self-recorded memory across successive rounds for self-learning, and it turned out that they achieved no noticeable improvement in those logs; their written memories were characterized by errors and excessive caution, trapping them in a repetitive defensive behavior.

This analysis carries direct practical implications for organizations and development teams in the Gulf, Egypt and the wider region, especially for entities aiming to move agents from chatbots to executors of automated processes in banking, logistics and government services. Success in the early steps of a financial-audit or transaction-review workflow never guarantees sound decision-making at later turning points, particularly as accounts accumulate. Relying on a textual memory recorded by the agent itself threatens to lock systems into overly cautious decisions that impede transaction flow, while the absence of clear inference poses an operational risk if the model encounters internal regulatory terms and classifications that differ from the familiar public data it was trained on.

“Spire Bench” provides a unsaturated benchmark that shows rapid efficiency assessments are no longer sufficient evidence of maturity in live work environments. A sound judgment of an AI agent is not tied to its ability to manage an isolated round, but to its capacity to allocate resources over a long trajectory, balancing today’s cost against tomorrow’s gains.