Beyond the perfect score: thinking logs revive saturated AI intelligence tests

AI intelligence measurement tests quickly reach a bottleneck when competing models achieve near-complete scores, causing metrics to lose their ability to distinguish between systems and hampering safety-evaluation decisions and advanced-model policies. However, a new research paper published by researchers led by Mario Hamada and Jeeyuk Yun via arXiv showed that exhausting accuracy scores does not mean the end of usefulness, because full solution logs, which document the detailed steps each model took to complete the task, retain rich signals that measure true capabilities even after the final numeric results have saturated.

The research team, which included Matteo Migliarini, Justin Oliv, and Magda Dupua, conducted a historical simulation of test saturation using earlier model versions and benchmarks, and they tested protocols that rely exclusively on correctly solved task texts to predict model performance in later unsaturated tests. The study compared three pathways for extracting signals: pairwise arbitration between model logs, absolute evaluation of each individual log, and prediction based on specified features and statistical embeddings.The results showed that pairwise arbitration clearly outperformed all methods, reducing predictive error by a factor of 5.3 compared with the traditional reliance on saturated test scores.

TheThe experimental analysis revealed that information indicating model efficiency is not evenly distributed across texts, but concentrates in specific channels, led by reasoning paths, thought chains, tool invocation, and external software calls. However, this mechanism has a methodological gap identified precisely by researchers: the predictive capability of protocols does not always hold against major structural shifts in model design.The experiment recorded a clear failure in assessing the efficiency of the first thinking and reasoning models that emerged in early 2025, as the records underestimated their true capabilities due to a difference in processing style from previous models.That means extracting the latent value from saturated tests requires flexible inspection tools that account for the architectural variability of systems and do not treat texts as a fixed template.

This disclosure has a direct impact on engineering and technology assessment teams in Gulf banks (banks in the Gulf region), government institutions and tech companies in Egypt and the Levant. Regional procurement agencies repeatedly face a dilemma when the offers of global vendors achieve success rates that approach 100 % in standard reference tests, which sometimes drives teams to exhaust large budgets building complex local tests from scratch. The practical benefit lies in shifting the testing compass from a pure accuracy percentage to an analysis of the execution logs themselves; comparing the number of tool invocations and the ordering of reasoning steps reveals the most efficient models and the least wasteful of computing resources without waiting for new testing cycles.

If you lead a technical team or oversee the evaluation of process models within your organization, your next step does not necessarily require abandoning the measurement standards that were nearing saturation, but rather examining what lies beyond the final result. The records ignored by traditional monitoring dashboards store the true differences in efficiency and operational safety, provided you build a paired arbitration protocol that measures the thinking and execution approach, while being cautious about applying the same old standards to any newly introduced model architecture.