Software agent architectures do not outperform models, and a controlled experiment overturns the hypothesis that developer-company frameworks have an advantage

A research study prepared by researcher Mohsen Arjmandi and published via arXiv reveals a precise experimental review of a common premise in AI-driven software engineering: that using a software framework developed by the model’s own creator gives the system a decisive advantage in solving programming problems compared with neutral frameworks. The software framework, also called the “sandbox,” refers to the toolkit, prompting formats, and control flows that turn a conversational model into an autonomous software engineer that inspects repositories and solves coding tasks.

The experiment was conducted through double-blind comparisons of the same model on a closed, contamination-free test set comprising 256 tasks distributed between real code repositories and competition problems prepared after the training data cutoff date.The researcher compared the original development kits with the neutral framework “Deep Agents” by executing 80 identical tasks.Model Claude Opus 4.8 was tested with Anthropic’s proprietary agent suite against the neutral framework, and model GPT 5.5 was tested with the OpenAI Codex suite against the same framework, with side-by-side versions of Gemini 3.5 Flash and DeepSeek-V 3.2 also included. A total of 792 out of 800 planned runs were evaluated by a fully isolated automated adjudication system to ensure scientific integrity.

The overall result refuted any statistically significant advantage for company-specific frameworks over the neutral framework. The Opus model recorded a slight negative gap of 1.25 percentage points in favor of the original package, with a success rate of 48.8 % versus 50.0 % for the neutral framework, while the GPT model showed a positive gap that did not exceed 1.25 percentage points, with a success rate of 55.6 % for the original package versus 54.4 % for the neutral framework. However, breaking down Opus’s results revealed an internal structural variance: the Anthropic original package fell behind by 9 percentage points on 61 real-world tasks in code repositories, yet it led by 23.7 percentage points on 19 algorithm-competition style problems, a gap the researcher notes was observed after data inspection and warrants an independent future study to confirm.

The measurements showed a serious disconnect between task completion time and final code correctness.Twenty-two of 81 workflows were cancelled for exceeding the preset time limit even though they had already produced successful code patches that passed automated testing, indicating that current agent-stop mechanisms waste viable solutions. On the cost side, re-pricing of token consumption showed that the neutral framework cost between 1.3 and 1.6 times the original framework per successful task on the Opus model, and 1.2 times on the GPT model, with the final financial ranking remaining unresolved because consumption logs were missing for 58 runs on Anthropic’s account, an issue this paper corrects after a remote-measurement error in its earlier draft.

This study places software-engineering teams in Riyadh, Dubai, Cairo and Amman before an operational reality that differs from the narrative promoted by major tech platforms. Rather than rushing to lock local technical stacks into closed packages from model providers on the claim that they understand the models better, the numbers demonstrate that building a neutral, flexible control framework gives companies the freedom to switch between models without losing productivity. The timing-out results also compel technology leaders to revisit the operational deadlines they grant their agents in production pipelines, as overly strict deadline enforcement may abort sound code fixes that could have been adopted.

The researcher made the automation code, adjudication system and analysis data publicly available while keeping the test tasks closed to prevent their leakage into future model training sets, providing a solid dataset for anyone seeking to separate intelligent model performance from the efficiency of its governing framework.