Haging Face warns of evaluation chaos, saying the linking tool outperforms leading models

AI labs build their reputation and commercial differentiation on standard evaluation leaderboards, treating test results as a neutral and precise metric that determines one product's superiority over another. However, the methodologies underlying these evaluations are complex and have not yet reached maturity, according to Irene Suleiman, head of policy at Haging Face, during a discussion at the TechCrunch Disrupt conference. Suleiman believes testing processes need to evolve into a coherent science if the sector wants to rely on them in the future, questioning the usefulness of the current metrics repeatedly issued by companies.

Suleiman places the concept of trust before a specific practical test: can developers and organizations reproduce the results of published evaluations? This capability is directly linked to the level of access available to the model itself, highlighting the importance of open weights as a fundamental pillar for verification and transparency. The lack of reproducibility remains a chronic problem in performance comparisons, compounded by the industry's inability to meaningfully compare different tests, as these evaluations quickly lose relevance in the absence of a thriving ecosystem that provides standardized criteria to determine who is qualified to evaluate for the required tasks.

Employing independent monitors inside AI labs is not an easy solution; it has become a defensive route to evade strict government regulations.This approach became widespread in recent months as labs faced a series of safety incidents that raised concerns about technology harms, and Haging Face itself was the target of one such incident. Labs turned to the idea of embedding neutral third parties within their facilities to avoid the need for active regulatory legislation, yet questions remain about the independence and expertise of these evaluators. The flaw became evident when Anthropic's decision to engage Accenture as its first evaluator sparked a wave of criticism and ridicule, following doubts about a general consulting firm's ability to conduct a rigorous technical review.

The challenge goes beyond external auditing to the core of technical architecture; tests shift from evaluating the model alone to measuring the entire surrounding system, specifically the harness and the smart agent. The harness, the means users rely on to access and employ the model, has a decisive impact on performance outcomes, to the extent that a less capable model can outperform the strongest leading models in practice when paired with an advanced harness. Evaluation is no longer confined to model weight alone, but to the quality of the software intermediary, and Suleiman notes that testing smart agents is extremely difficult and costly, and the industry still lacks sufficient assessments of harness packages and integrated agents, even though the real value for users derives from this holistic system.

The operational value of intelligent systems is shaped by the quality of internal harness tools, not by theoretical leaderboard results.This reality carries strategic implications for tech teams and organizations in the Gulf, Egypt, and the region, especially banks and service entities that allocate massive budgets to contract with closed interfaces of leading models. Naïvely relying on Western evaluation tables to purchase expensive access to the latest models without comparable investment in harness and deployment engineering is a misdirected expense; building a polished software stack around an open-weight model could give an organization genuine advantage at lower operating cost. Moreover, regional institutions' turn to traditional management consulting firms to assess AI solutions repeats the failures of international markets, necessitating the development of software audit talent that understands agent architecture and tests outputs in live production environments.

The AI evaluation landscape is characterized by randomness and a lack of reference standards, rendering current market comparisons unsuitable for decisive decision-making. Preserving measurement relevance requires building a unified scientific pathway that spans the entire industry, shifting focus from abstract model scores to examining the comprehensive interactive system on which organizations base their real-world operations.