OpenFrontiers reveals the myth of scientific discovery: seven agents are powerless against closed memory
New academic research draws clear limits to the growing claims about AI agents' ability to achieve independent scientific discoveries, revealing that the commonly observed superiority in solving mathematical puzzles sharply declines as soon as models are isolated from the open internet. A new research paper titled “OpenFrontiers”, deposited by researchers Jiariui Yao, Roida Wang, and Tong Zhang on the arXiv platform in late October, tracked language models' ability to rediscover science using an evaluation metric that combines open problems with those solved recently.
The team built an extensive database containing 12,151 open research problems across multiple scientific disciplines, and the researchers documented the problem statements, their theoretical context, and the trajectory of knowledge progress achieved. From this repository, the researchers derived a benchmark consisting of 200 mathematical puzzles; of these, 110 problems were solved during 2026, and 90 other problems remained open without solution as of the paper's preparation.To ensure the integrity of mathematical verification, the researchers expressed each problem in the formal proof language Lean 4 and subjected them to a double review that included software probes and an independent examination.
The researchers did not stop at collecting and documenting the problems; they also designed a precise evaluation protocol based on a high-quality sub-suite of 50 problems selected according to qualitative evaluation criteria. The team tested seven agents built on large language models under a uniform computational budget, and the resulting proofs underwent double independent scrutiny within a completely isolated software environment. The experiment’s assessments were divided into three tracks: a closed-memory track relying entirely on the model’s internal weights, a frozen-retrieval track from a closed database, and a live-connection track to the internet.
The results showed a structural discrepancy between mental reasoning and digital indexing; in the closed-memory track, the agents’ success at rediscovering solutions was extremely rare, with the best models solving only seven problems out of the forty solved problems included in the evaluation.The experiment also revealed that model superiority in a live-web environment does not reflect genuine inferential ability, but rather is largely tied to retrieving solution texts that human researchers have published online.This gap indicates that current models’ ability to produce new mathematical proofs remains constrained by the existence of prior solution paths.
This shift is linked to the nature of technical decisions made by research labs and engineering development teams in the Gulf, Egypt, and the Arab Levant, especially as the deployment of intelligent agents to automate scientific-research tasks expands. Relying on models to carry out precise mathematical or software derivations based on their performance in open environments entails a mis-estimation of their intrinsic capabilities, raising the risk of consuming computational resources on solutions that amount to merely re-arranging what has already been written.What this evaluation requires of developers in the region is shifting testing standards from superficial metrics to rigorously programmed, constrained examination environments.
These findings compel technology managers and researchers in regional centers to shift skill focus from textual prompting engineering to formal proof engineering using mathematical verification languages. Employing artificial intelligence in infrastructure projects or economic and sovereign modeling mandates testing systems within closed environments that block external research, to verify that the model can reason logically rather than perform automated retrieval. The decisive factor for adopting scientific outputs remains disciplined engineering verification of each step, rather than relying solely on glossy solutions sourced from search engines.