Real-SWE benchmark tests programming agents in production environments: success rates fall to 38% on complex business rules

Specific Labs launched the software evaluation benchmark “Real-SWE”, designed to measure the efficiency of leading AI models when dealing with proprietary, production-grade codebases that have been licensed from actual technology companies. The step aims to address a long-standing gap in traditional evaluation metrics, placing models and programming agents in front of real work tasks previously handled by salaried software engineers, rather than relying on synthetic problems or open-source repositories that can easily leak into the models’ pre-training data.

The benchmark’s results, which used the average of eight independent attempts per task measured by first-attempt accuracy, revealed a clear decline in agents’ ability to execute code changes accurately compared with synthetic tests.Model “Viper 5.1” running on the “Cloud Code” environment topped the performance list with a solution rate of 38.8 percent, followed by “GPT-6 Astra” via the “Codex” interface at 33.8 percent.The “Gemini 3.8 Flash” model via the “Gemini” interface placed third with 31.2 percent, ahead of “GLM 5.3” at 28.8 percent, while “Groq 4.6” and “Muse Spark 1.3” tied at 23.8 percent, “Cemi K 3” held steady at 18.8 percent, and “GPT-5.6 Sol” at 16.2 percent.

The test data showed that the core difficulty lies in shared-environment requirements and the interdependence of external services. The benchmark’s average instruction length is 1,742 characters, and a typical solution modifies an average of 11 files in a single workflow, compared with six files in benchmarks such as “Frontier Code” and “Deep S DW”. Task analysis also indicated that six of the ten sampled tasks recorded overall success rates below 15 percent, that analysis-flow-reduction tasks achieved a zero-percent success rate across all models without exception, while state-tax-calculation tasks rose to 3.1 percent and linear-consistency scans of data reached 4.7 percent.

Failure classifications show that overlooking detailed requirements and basing decisions on unverified assumptions constitute the biggest obstacle to autonomous reliance on programming agents.The failure rate for coding rounds lasting under ten minutes reached 71.4 percent, while 73.4 percent of longer rounds failed, indicating that additional thinking and inference time does not compensate for a lack of precise understanding of system architecture and its accounting or operational rules.

The benchmark reshapes the calculations of technology leaders and development teams in banking institutions and digital platforms across the Gulf and Egypt, where the delegation of programming agents is expanding to accelerate banking services and electronic payments development. Relying on high success rates in open repositories creates a false sense of operational safety; the more local systems are tied to tax constraints or multi-party financial-settlement regulations, the higher the likelihood of generating misleading code that ignores verification conditions or assumes incorrect external-service behavior, prompting a shift in selection criteria from raw model speed to the rigor of verification tools and integration testing before any output is merged into actual production.

The Real-SWE benchmark demonstrates that the shift from programmer assistance to full autonomous operation in private work environments still encounters genuine structural constraints, and that enterprise software engineering goes beyond function definition to understanding the system’s historical context and its direct financial and operational ramifications.