The “Ansatz” benchmark reveals a gap in scientific discovery: AI agents detect variables but fail to formulate their laws

A physicist sets their core hypothesis before adjusting any numerical coefficient in an equation, defining what appears as quantities and what disappears into the shadow, how variables intertwine and are measured, this foundational step is called “Ansatz” by scientists. In a new research paper led by Ethan Ji, Wei Wang, and Yichuo Sun and posted on the arXiv platform, large language models were subjected to a test that probes the heart of scientific reasoning using a novel benchmark the team named “Ansatz.” The crucial difference is that earlier benchmarks for automatically discovering mathematical equations presented this structure on a ready-made plate by specifying variables and a list of available bounds, whereas the new benchmark forces the AI agent to construct the structural hypothesis itself from scratch.

Formulating the abstract mathematical hypothesis is the real bottleneck in scientific discovery, not merely estimating numerical coefficients.To test this ability, the researchers created eighty simulated systems spanning twenty different scientific domains, ranging from ordinary and partial differential equations with time delays to discrete maps, causal models, and probabilistic processes, and equipped them with hidden states, distorted probes, and uninformative measurement channels to mimic the complexities of physical reality. The AI agent operates within the benchmark by purchasing experiments within a fixed budget, then writing candidate programs in a miniature modeling language, receiving numerical parameters from a dedicated matching tool, while its performance is measured solely by the final program’s ability to predict experiments it has never seen, including conditions outside the explored space, experimental interventions, and longer time trajectories.

The results revealed a clear spread among the nine language models tested, with success rates ranging from 21 % to 60 % in solving these systems. The most striking paradox involved forty-four systems that classical equation-discovery methods could not solve at all, even though the aggregated data were sufficient to compute the coefficients once the model was supplied with the correct equation form, demonstrating that the real difficulty lies in uncovering the structural framework rather than estimating its numbers. When researchers tried to feed the models with programs previously discovered for similar systems, it made no positive difference unless the underlying mechanism was already shared, in which case the first discovery succeeded.

Behavioral analysis of the agents showed two decisive phenomena that reveal the nature of current algorithmic thinking: stronger models make fewer scientific errors, yet models in general succeed at identifying the variables present in a system while failing to grasp how those variables interact physically. More concerning is the agents’ tendency to fall into a computational bias, as experiments showed they prefer to tweak equations in ways that reduce training-error margins on the data rather than choosing modifications that improve future predictive accuracy, a behavior that departs sharply from rigorous scientific methodology in favor of superficial number-matching.

The models’ bias toward minimizing training error at the expense of predictive accuracy calls for a radical reassessment of modeling projects in the region.For those leading computing and research teams in Gulf labs and research centers in Saudi Arabia, the UAE, and Egypt, especially groups working on energy-site simulations, power-grid modeling, or desalination plants, this benchmark offers a crucial practical lesson. You cannot rely on AI agents to craft independent physical models from data streams alone; the models will produce equations that appear mathematically identical to your historical records but will collapse when faced with sudden operational changes. The sound technical decision is to task human engineers and researchers with defining the Ansatz and structural physical constraints, limiting the models to constrained exploration and proposing the coefficients.

The study concludes that scientific discovery does not start with gathering numbers but with envisioning the hidden relationships that bind them, a skill that AI models are still only beginning to grasp and likely will not master until the end of 2026. The next challenge for R&D teams is not faster computing but training models to resist the lure of superficial number-matching and to adopt scientific abstraction capable of withstanding unseen field tests.