CordisBench tests AI agents in managing their code: high cost and failure to track dependencies

A new research paper by Damian Celio and Dmitri Kascher tests language models' ability to manage software lifecycles autonomously by introducing an evaluation metric called CordisBench. The study focuses on what are known as dynamic agent execution frameworks, software environments in which a language model is granted permission to modify the code and attachments that govern its own execution. This mode of self-execution places a complex inference burden on the model, because any local change in a software attachment may extend its effect through dependency chains, cleanup steps, and background component shutdowns.

The CordisBench metric comprises 1,200 questions designed to test this type of computational reasoning, combining a disciplined visual framework with programs that actually run inside an execution environment called Cordis, a platform dedicated to managing component dependencies and their termination and cleanup processes. The evaluation tasks are distributed across four defined tracks: identifying the software components affected by any modification; predicting the system state after a specified dismantling sequence; determining the conditions that remain true under all possible orderings, or under some of them; and selecting reinitialization plans that actually succeed when executed.

The researchers subjected three models optimized for operational efficiency to the test with low inference effort, measuring performance across six levels of relevant program interactions that start with two interactions and scale to 4, 8, 16, 24, and finally 32 interactions, depending on deterministic logging mechanisms customized for each task. The results show that the models succeed in handling small software systems with limited interactions, but their reliability declines sharply as the interaction network becomes more complex, and this failure becomes evident when attempting to predict the system's final state and when reasoning about multiple dismantling and shutdown paths.

Increasing inference effort gives some models noticeable jumps in prediction accuracy, but these gains come at a heavy operational cost; in a test set containing 16 interactions, the GPT 5.6 “Luna” model consumed roughly 3,000 thought tokens per question when run at a medium effort level.The study also reveals that this high computational expenditure can be avoided entirely, after the results demonstrated that constructing ultra-fine, independent attribution semantics at the code-structure level matches the execution of the Cordis environment in every observation and outcome across 528 fully executable questions, meaning that deterministic sequencing of software operations outperforms burning generative tokens to simulate program logic.

This discovery carries a highly significant practical dimension for engineers and development teams in the Gulf, Egypt and the Levant who are building AI agents to manage complex cloud systems and services. Relying entirely on a generative model to track component lifecycles and compute closure and dependency paths raises inference costs enormously and leaves the system unable to guarantee stability once the number of interactions reaches advanced levels. The prudent engineering remedy is a strict separation in the infrastructure: assign lifecycle management and consequence calculation to a rule-based deterministic software layer, and keep the language model confined to high-level guidance and coordination tasks, thereby protecting operational budgets from depletion and preventing the collapse of automated workflow paths.