BenchMIRT breaks through generative model tests, showing psychometric measurement reveals safety overlapping with reasoning

The aggregate numbers that top language model evaluation leaderboards reveal recurring contradictions, as they sometimes fail to explain why a model excels in one real-world setting and falls short in another. The Allen Institute for AI introduced a new tool called BenchMIRT, borrowing psychometric methods to analyze AI intelligence tests at the level of individual questions and single tasks, instead of relying on overall scores that mix disparate abilities within the same test.

The tool relies on Multidimensional Item Response Theory (MIRT), a statistical methodology that originated in standard psychometrics to identify traits and mental abilities through response patterns, because each question does not provide the same amount of information about the examinee's ability. Researchers applied this approach to the results of one hundred open-weight language models across sixteen benchmark tests and more than 34 000 questions, without giving the tool any prior guidance about what each test measures. The mathematicalThe tool relies on Multidimensional Item Response Theory (MIRT), a statistical methodology that originated in standard psychometrics to identify traits and mental abilities through response patterns, because each question does not provide the same amount of information about the examinee's ability. Researchers applied this approach to the results of one hundred open-weight language models across sixteen benchmark tests and more than 34 000 questions, without giving the tool any prior guidance about what each test measures. The mathematical analysis produced two primary independent dimensions of performance across all questions: a general inference dimension and a safety dimension.

Deconstructing common tests shows that the stated evaluation goals do not always align with what the question actually measures. The “BBQ” test, usually classified among bias and social safety assessments, was linked in the analysis to general inference more than to safety; the question, akin to choosing a taxi for a grandchild versus a grandfather, requires tracing logical evidence and the sequence of events rather than merely demonstrating sensitivity to age. In the “WMDP” test, designed to measure dangerous dual-knowledge concealment in chemistry, biology and cybersecurity, a reverse correlation emerged between higher model inference ability and lower scores, because success on the test requires refraining from providing the information, which penalizes a model that has a greater capacity to give complete answers.

Decomposing the individual questions shows that the safety assessment may measure understanding of the issue rather than a tendency to comply with the controls, which changes the reading of the reference board results.

The contrast extends to protection-breaking tests such as Harm Bench; while malware and fraud extraction questions reflect actual security after the fact, intellectual-property questions, such as requesting the lyrics of a popular song, are linked to general reasoning. The tool allows the question banks to be reordered according to their discriminative power, and experiments have shown that retaining only 10 percent of the test questions is sufficient to obtain the same measurement of the model’s capabilities in safety and reasoning, with the ability to predict the models’ answers to hidden questions with an accuracy of 79 percent compared with roughly 70 percent for the traditional reliance on the average test score.

This shift has a direct practical impact on AI engineering teams and chief technology officers in banks and government agencies across the Gulf, Egypt and the Levant, as it ends naïve reliance on digital averages when selecting or evaluating open-source models before operational integration. It also allows test sets to be reduced to one tenth of their size, cutting computing costs and inference time needed to evaluate models locally, which lowers operational verification budgets for startups and regional research labs. However, this transparency also carries technical caveats, as detailed metrics may be used to remove the most stringent questions and engineer incomplete tests that unprotected models can pass.

The results of Bunch Merit remain constrained by the models they were trained on and the time frames extending to early 2025, but they set a new benchmark for designing upcoming evaluation tools and underscore the need to examine the questions themselves to understand what they truly measure before making software release decisions.