Model evaluation disparity doubles outside their weights: a study identifies eight components of the agent operating framework

Large language models do not possess the ability to act independently; they operate solely through the surrounding infrastructure, known as the operating framework or agent host. This surrounding framework practically determines what the model can achieve and where it fails, yet it remains, to date, without a unified, stable standard definition in technical and research circles. In a new research paper submitted to the arXiv archive on October 29, researchers Nicolas Gertler, Claudio Novelli, and Luchiano Floridi provide a systematic dissection of the intelligent agent operating framework into eight functional components that can be described and compared across different systems, proposing a transferable standard framework that accurately records the model used and the precise structure it operated within.

This step gains its urgent necessity from an engineering reality that many assessors overlook:Changing the configuration and tuning of the operating framework can lead to differences that multiply several times in the recorded test results of the same model.Measuring a model’s efficiency in isolation from its execution environment and tools gives developers and business teams misleading results, because the numeric outcome reflects the integration of the surrounding framework more than the strength of the model’s weights. The crisis deepens as cyber-threat spaces expand, with the study confirming that most of the attack surface threatening intelligent agents lies in their operating frameworks rather than in the abstract core models.

The study illustrates the shortcomings of current documentation tools, such as model cards and agent cards, which provide only general release and operation data but fail to track the chronological sequence of actual configurations within a single execution session. Frameworks change from one release to another, even between sessions, reaching peak complexity when an agent can modify its tools and settings autonomously during task execution. Although the AI regulations that have come into force impose documentation obligations on the systems covered, they do not require developers to maintain a unified record of operating frameworks, prompting researchers to demand that the operating framework be treated as a describable, version-numbered component like any major software element.

This proposal goes beyond academic debate to directly impact engineering teams and technology managers in the Gulf, Egypt, and the Levant.When banks, government institutions, and start-ups in Riyadh, Abu Dhabi, and Cairo begin building automation agents for customer service or operations management, the decision often leans toward purchasing model licenses based on global leaderboards, assuming that security and efficiency stem solely from the model. The study’s findings practically mean a need to redirect skills and spending: real testing requires building and documenting a robust operating framework that controls tools, permissions, and access limits, rather than relying on model weights alone. Moreover, institutions’ adherence to regionally issued data governance regulations will soon require proof of the operational host’s integrity to avoid permission leaks and breaches, making the engineering of operating frameworks a critical specialty equal in importance to command engineering or model hosting.

The shift toward autonomous agent automation demands abandoning the treatment of models as isolated software endpoints and viewing the technical host as the entity actually responsible for productivity and security limits. Documenting every modification an agent makes to its tools within its operating environment is no longer a research luxury but a baseline for risk assessment, failure analysis, and building enterprise systems that can be legally and operationally audited.