CARAT test reveals the illusion of material model inference: text memorization overrides understanding of crystal structures

A research paper authored by Jiajun Wu and his team, published on the arXiv platform under number 2609.38340, subjects language models for materials science to a rigorous systematic test: do these models truly infer from geometric structure and crystal lattice, or do they merely copy a pre-written answer embedded in the input text? The paper reveals a core problem: conventional accuracy metrics fail to uncover the truth, because the structural description in prevailing experiments often reproduces the same field on which the model is evaluated.

To address this shortcoming, the researchers designed an evaluation framework they named CARAT, which relies on fixing the question and the reference answer across eight identical presentation modes, labeling each structural relation separately within a GraphSpace representation, and integrating matched fine-tuning techniques, answer masking, evidence injection, pairwise reasoning, and a rule capable of suppressing claims. The results showed that the structure-aware presentation achieved a 17.3-point advantage over inputs limited to abstract formulas in the most challenging test families. However, the researchers subjected the results to a self-audit that demonstrated thatthe superiority of accuracy metrics does not reflect reasoning quality, it turned out that GraphSpace’s improvement over a simple periodic diagram was 19.3 points, split into two contradictory effects: the gap did not exceed 1.96 points when the simple presentation contained everything the question required, but jumped to 46.7 points when the simple presentation omitted those fields entirely, confirming that high numbers mostly measure baseline deficiencies rather than how evidence is presented.

The evaluation probed the test itself, and the team discovered that a rudimentary code rule that bypasses the link and reads the list directly could answer four out of seven families in the hardened test, prompting a reconstruction of the system until eleven such looping gaps fell to the level of chance. When the frozen model was tested without subsequent training, a behavior emerged that shows thatthe model repeats structural links without employing them, as it copied the link verbatim but continued to give the same answer even after its direction changed in 95.6 % of paired cases, repeating the relation without any inferential thinking. In contrast, matched supervision and training raised usage accuracy to 99.8 %, and when the link was removed performance fell to 23.4 %, below the 27.0 % achieved by the best looping rule, demonstrating that the ability to move from memorization to deployment can be acquired through learning.

This analysis is highly relevant for research teams, computational chemistry labs, and advanced materials discovery groups in the Gulf and Egypt, especially for entities that rely on AI to advance energy, battery, and water-treatment research. Purchasing compute subscriptions or training frozen models without subjecting them to matched-presentation tests and looping-gap checks means spending budgets on language models that give their users the illusion of advanced reasoning capabilities while they merely reproduce what was fed in the inputs. The practical change required from research-center leaders in the region is to revise internal testing protocols and enforce data-masking methods to ensure that developed models grasp the physical properties of compounds rather than merely retrieving text.

The CARAT experiment shows that the true benchmark for AI model efficiency in scientific research is analytical ability rather than memorization. If you lead a technical or scientific team in your organization, your first step is to dismantle generic accuracy tests and scrutinize baselines carefully, because superficial digital superiority may conceal replication that fails to deliver genuine knowledge value.