OmniScientist: a multimedia artificial intelligence system driving the research cycle from raw evidence

A new research paper deposited by computer science and language researchers on the arXiv platform reveals the OmniScientist system, an integrated framework that constitutes a comprehensive artificial world for media and across disciplines, designed to conduct scientific research directly from heterogeneous raw evidence and data. The paper, prepared by Bobo Li, Hao Fei, Tianji Gu, Mong-Lee Li, and Win Hsu, shows that recent advances in foundational models have succeeded in automating increasingly complete research workflows, including hypothesis generation, code execution, and draft preparation, but covering the workflow alone did not provide access to the full set of evidence on which scientific discovery relies.Previous systems have traditionally relied on inference through texts, source code, ready-made labels or precomputed summaries, which has made spatial, temporal, procedural and cross-channel relationships unavailable to the intelligent agent during its operation.

The proposed system is built on an integrated architecture that spans from start to finish, comprising a perception layer and three independent agents: the first is dedicated to idea generation, the second to experiment management, and the third to writing and drafting scientific papers. This team of agents operates within a deterministic pipeline, allowing direct sensory observations derived from data to guide the formulation of research questions, determine the path of experimental decisions, and construct final claims across the various stages of the scientific research cycle. To support scientific quality and avoid inaccurate conclusions, the system incorporates software verification mechanisms that include checks of ideas, methodological accuracy, and claim validity in the code, which in practice requires filtering ideas to ensure their novelty, verifying the statistical validity of experiments, documenting the provenance of execution, and ensuring digital traceability of all derived results.

The research team subjected the Omni Scientist system to an extensive experimental evaluation that included 36 case studies based on real and authentic data, and these cases were distributed across five families of scientific disciplines and four main families of scientific evidence. The media the system handled varied to include images, signals, audio recordings, video clips, three-dimensional spatial structures, motion paths, digital tables, mathematical formulas and complex graphs.The evaluation results showed that the system was able to complete the entire research workflow, from raw data to a compiled draft research paper, in all thirty-six cases.Achieving an average overall rating for the resulting papers of 6.3 points when using the specified reference inference model in the experiment.

To assess the effectiveness of direct data perception, the researchers conducted direct paired comparisons between the system and a stripped-down version designed to receive only precomputed numerical features. The result confirmed that providing direct perception of evidence improved performance across all seven evaluation dimensions used in the study, with the perceptive version winning 85 % of the direct vertical evaluation rounds. The paper, which spans thirty pages and includes thirteen illustrative figures and nineteen tables, concludes that making sensory perception of evidence available throughout the entire research cycle is a decisive factor for grounding a scientific discovery in reality and paves the practical way for building artificial-intelligence agents that possess broad, multidisciplinary scientific research capabilities.