The “Baker” benchmark reveals multimodal models’ inability to manage autonomous chemical labs

Autonomous chemistry labs are gaining rapid momentum as a new interface for automating scientific research, yet the bet on large multimodal language models to serve as embodied minds that run physical experiments faces a sharp operational gap. A new specialist evaluation, accepted at the NeurIPS 2026 conference in the evaluations and datasets track and selected for a featured presentation, showed that these models' ability to grasp experimental outlines does not equate to the ability to execute them in the lab. The benchmark, named “Baker,” for the first time sets a strict metric for embodied actions, knowledge and experimental inference, revealing that the models are unable to translate theoretical understanding into precise motor decisions on the interaction table.

Fully autonomous labs require the chemical experiment to shift toward embodied automation, where machines must perceive laboratory scenes, analyse reaction states and operate under precise physical, procedural and safety constraints. Although multimodal models appear as a leading candidate for these tasks, previous evaluation standards have mostly focused on general safety, anomaly detection or abstract scientific understanding, leaving embodied execution capabilities unexamined. To fill this gap, a research team including Filin, Tingchao Zhang, Xiuye Yang and fifteen other researchers developed the “Baker” benchmark, comprising one thousand question-answer samples crafted entirely by chemistry and embodied AI experts across eleven categories of image and video tasks, without any generated or automatically verified content.

The study examined twenty-eight multimodal models.They span general-purpose models, science-oriented systems, chemistry-specific models and embodied AI models, all evaluated in an offline, single-step testing environment. The results draw clear boundaries for these systems: the models can track general laboratory procedures shown in video clips, but they fail dramatically when it comes to projecting actions onto directly operable regions and identifying physically executable motion paths. They also struggled to accurately recognise detailed lab tools and pinpoint reaction states, and they faltered in adhering to safety rules and ordering steps within medium-length protocols.

This failure sets critical limits for research centers and industrial complexes.In the Gulf, Egypt and the broader Arab region, where investments in automating petrochemical, polymer and drug-discovery labs are rising, the costliest mistake digital-transformation and industrial-development managers could make is conflating the models' textual fluency in reading chemical papers with their suitability for controlling equipment and handling reagents. The reality of these findings forces a reassessment of engineering standards, as designing smart labs now requires limiting models to constrained cognitive planning while assigning actual execution to robotic systems governed by hard rules and direct human supervision, to avoid chemical-incident hazards and costly equipment damage.

The diagnostic benchmark results confirm that the path to full autonomy for chemistry labs does not go through conventional multimodal models unless the challenge of motor perception linked to physical actions is addressed.Bridging the gap between visual understanding and tangible executionremains the primary condition before allowing intelligent software to move beyond generation screens to managing live experimental equipment.