Experiment on 102 scientific studies sets the ceiling for artificial intelligence: broad discovery and inability for critical synthesis
Scientific literature reviews usually start with the hard question: how can one capture every study related to the research topic and separate the chaff from the wheat without missing a seminal paper? A new research paper led by researcher Abdullah Al-Kalbani with four co-authors, published on the Arkhive platform, offers a decisive experimental answer that examines the ability of large language models to manage this process across four consecutive stages. The authors used a published scientific review of the “Spanish column” pattern and its reference set of 102 scholarly papers as a test benchmark, to compare the performance of six AI-based tools in searching, interpreting, synthesising and proposing future research pathways.
The initial search and exploration phase revealed a sharp gap between open access and restricted access via academic databases; the six tools together found only 57 studies out of 102, while the best single tool stopped at just 42 papers.This picture changes fundamentally as soon as the tools are linked to institutional library subscriptionswhere the tally for the “Codex” tool jumped from 42 to 82 papers, and the results for “Cloud Code” rose from 15 papers to 79 papers. This leap highlights that the main obstacle to using models for scoping scientific literature does not lie solely in prompting or reasoning efficiency, but in paywalls and closed databases that hide peer-reviewed research from open-access retrieval engines.
When the tools were supplied with the full reference set in PDF format to test their comprehension and classification abilities, three tools succeeded in identifying the eighteen studies that addressed the phenomenon in a substantive and detailed way. However, the snag became evident in classifying the remaining eighty-four papers, which contained only passing mentions; the models classified between 40 % and 79 % of those passing mentions as core contributions. This confusion creates an entire research gap, risking the construction of conclusions on marginal citations that do not constitute a genuine scientific weight in the core of the specialized issue.
The experiment’s shortcomings did not stop at classification; they extended to the scientific synthesis stage, which was evaluated in a blind test by two experts in the field. The results showed that the summaries produced by the three tools, while generally accurate, lacked deep analytical insights, contained unsupported claims and citation-percentage errors, as well asoverstating a scientific consensus that does not exist in the actual literatureWhen the experiment asked for suggestions for future research directions, the models offered vague, generic proposals that went beyond the bounds of the studied phenomenon’s literature, with two tools approaching questions flagged by the original standardised review.
These conclusions acquire exceptional relevance for research communities and development labs in the Gulf, Egypt and the Levant, where reliance on AI tools to shorten the time needed for white papers, feasibility studies and academic reviews is growing. The practical outcome obliges universities and research centres not to rely on free-tier or open-access model queries, because they miss more than half of the published scientific output and make institutional-database integration a prerequisite for any rigorous work. The findings on passing mentions and the models’ tendency to fabricate a false scientific consensus also mandate prohibiting the delegation of final synthesis to intelligent tools without strict human verification by a specialist researcher who extracts citations and checks the validity of the evidence.
The study issued eight guiding recommendations for researchers, concluding that large language models can accelerate discovery and facilitate initial reading of scientific literature when connected to appropriate repositories, but they are wholly incapable of replacing the critical judgement of a human expert. If you lead a research team or are preparing a systematic review, let this experiment be your practical guide: use models to scan the horizon and retrieve texts, but retain the authority for filtering, evaluating and final synthesis within your own human mind.