TraceML study explores why software agents fail to match experts in machine learning engineering

Large language models can write precise code snippets when tackling isolated or narrow problems, yet they remain unable to independently manage the full development cycle in machine learning projects. This shortcoming becomes evident when software agents face work environments that demand continuous hours of feedback, data pipeline adjustments, model experimentation, and validation tuning, where they ultimately lag behind seasoned human competitors in most contests.

A new research paper titled TraceML, authored by researchers led by Jiarui Yan, shows that traditional outcome-only evaluation metrics capture this human-machine gap without uncovering its root causes, as they judge only the final submission and overlook the entire development trajectory. To address this limitation, the researchers introduced a comparative framework that collects and documents development trajectories at the level of granular code versions and revisions, benchmarking human developers against software agents in the same Kaggle competitions.

The dataset encompassed 4,465 human development trajectories across 134 competitions, including seven contests run in parallel using two agent architectures, enabling the analysis of 430 human trajectories against 207 agent trajectories. The study documented each code revision alongside its achieved score, timestamp, action classification, objective, edit size, and immediate impact on performance.

The fundamental divide lies in the human engineer's ability to pivot mentally and revisit past experiments, whereas agents fall into narrow and unproductive development loops.The analysis shows that human experts follow an agile trajectory that balances data engineering, validation tuning, switching model architectures, and ensembling predictions, while retaining the flexibility to backtrack to previously discarded ideas and paths to retest and refine them.

In contrast, the study revealed that software agents confine themselves to narrow, repetitive patterns. The Codex architecture exhausts its steps in reweighting ensemble models and fine-tuning submission details, while the ML-Evolve architecture limits itself to localized tweaks to the existing model without structural changes. Both architectures fail to change direction at the pace humans do, and they prove completely incapable of reopening abandoned development paths.

When the researchers supplied the models with a planning prompt derived from human expert practices, agent behaviors improved in explicitly targeted areas and their scores rose, yet the overall structure of their effort remained constrained by the agent's narrow nature. The experiment demonstrated that prompts and text instructions bridge only that portion of the gap reducible to direct commands, while agile planning and exploration remain beyond current reach.

This analysis carries direct implications for data and AI teams across the region, whether in tech hubs in the Gulf or software firms in Egypt and the Levant. Relying on off-the-shelf software agents to build machine learning models without ongoing engineering oversight leads to dead ends and wasteful compute expenditure on superficial experiments. This requires team leads to prioritize planning, validation, and pipeline design skills as critical human competencies that automated generative prompts cannot replace.