SWE Refactor test shows programming agents fail to migrate full software packages with success rates no higher than 5.4%

A new research paper published by researchers on the ArXiv platform reveals a structural gap between AI models' ability to fix limited programming errors and their ability to migrate and update complete software repository packages and eliminate accumulated technical debt spanning decades. The paper, which introduced an evaluation benchmark called "SWE Refactor Bench", tests whether independent code agents are actually capable of performing complex, long-term migration operations that encompass entire code repositories across four types of technical debt distributed over twenty real migration tasks.

The study highlighted a fundamental shortcoming in previous evaluation metrics, which only verified functional behavior without checking whether migration had actually occurred. This gap opened the door to what the researchers called a “blind” situation, where the code agent simply copies the old original code unchanged to ensure passing behavioral tests without altering the required technical architecture. To address this flaw, the team designed a three-stage evaluation protocol: the first stage audits the migration to confirm structural change, followed by a stage of static behavioral tests to assess accuracy, and concluding with an agent verification stage that employs six independent code agents to generate custom tests that reveal subtle, hidden behavioral differences.

The quantitative results were decisive in showing the current leading models' inability to independently manage full migration.Of the 520 run experiments involving eight advanced models across 26 different effort-level configurations, only 28 experiments passed all three stages, yielding an overall success rate that did not exceed 5.4 percent, while all models without exception failed to provide any acceptable solution in 13 of the twenty tasks. The top-performing model, Claude Opus 5, did not surpass a score of 47.0 out of one hundred, reflecting the difficulty of maintaining behavioral consistency while performing full structural migration.

The data also showed a sharp variance in agent efficiency depending on task type, with models averaging 31.4 points on build-tool and compilation-chain rewriting tasks, while performance collapsed to only 5.6 points on full programming-language rewriting tasks. The study demonstrated the difficulty of delivering a flawless migration: of the 340 experiments that passed the structural migration audit, 58 percent reached 99 percent of the static checks, but only 26 percent achieved a full completion mark of one hundred percent.

These figures set clear limits for any plan that relies on full automation to update legacy software systems.For technical managers and development teams in banking institutions, government agencies, and tech companies across the Gulf, Egypt, and the region, these results mean that relying on code agents to migrate legacy databases or rewrite core systems in modern languages without intensive human oversight carries high operational risk, as the remaining hidden errors are sufficient to disrupt sensitive production environments, making investment in auditing and human review platforms a mandatory prerequisite before adopting any automatically generated code.