Open models break cyber-security monopoly, achieving effectiveness comparable to the major players at one-tenth the operational cost

Institutions seeking to employ AI agents in security tasks face a complex trade off between open weight models that allow self inspection and autonomous operation, and the more expensive closed models.An extensive benchmark study on the CTI-REALM platform shows that open models can match the strongest closed models at less than one tenth of their cost, provided they are given a sufficient number of interaction rounds.This conclusion, documented by researchers Sahil Jarg, Rohi Gotomokala, Vishal Agarwal, and Shubham Vishwakarma in a new research paper, redefines the economic and technical viability of building automated defense systems within cyber operation centers.

The CTI-REALM benchmark measures the linguistic agent’s efficiency in converting threat intelligence reports into operational detection rules by issuing direct queries to a remote measurement repository. The team evaluated twenty six models, nineteen open weight models and seven closed models, under a unified testing environment, a consistent evaluation tool, and a judging panel based on language models. The experiment revealed a structural flaw in the previously published benchmark; it imposed a hard cap on the number of agent messages, and any model that exhausted this cap received a complete zero on all verification points, even those it had previously passed.

This prior constraint caused the measurement to focus on completion speed instead of actual detection accuracy, obscuring the true differences among the leaders.When the researchers removed this arbitrary constraint, the open models regained their balance and progressed to achieve the highest reward scores in the entire test.The results demonstrate that price does not reflect efficiency in any case; models that earned comparable rewards had operating cost bills differing by more than an entire order of magnitude, i.e., by over ten times the cost.

We examined at AI Gossips the details of the experiment, which did not stop at comparing cost and overall results but extended to the internal architecture of command generation; researchers measured the token volume produced by each model and found that the portion of output allocated to internal reasoning does not correlate with higher reward accuracy or detection effectiveness. Additionally, the team replaced the traditional verification points, which merely recorded what the agent mentioned in conversation, with staged metrics that examine what it actually delivers. The detection queries were executed directly within the data repository and their outputs compared against the approved answer keys, separating true technical performance from abstract linguistic claims.

This staged measurement revealed a decisive engineering insight: no single model dominates all workflow stages. One model may excel at analyzing threat text and extracting indicators, while another leads in formulating complex measurement queries and validating output integrity. This outcome directly dismantles the notion of an all purpose model and guides software architects toward a staged orchestration approach, assigning each sub stage of the workflow to the model that performs best in that segment rather than coupling the entire system to a single engine based on its overall rating.

This shift touches the core of security work in banks, government agencies, and data centers across the Gulf, Egypt, and the Levant. These environments are subject to strict regulatory requirements that mandate protecting data sovereignty and preventing leakage of communication logs and remote measurements beyond borders, which previously conflicted with the high inference cost of proprietary models or the difficulty of programmatically auditing them. When a locally deployed open model operates within an enterprise’s infrastructure and delivers accuracy comparable to global vendor interfaces at less than one tenth of the consumption cost, cyber security budget calculations change entirely.

If you lead an engineering team in a regional defensive operations center, your practical step requires stopping the pursuit of closed subscriptions for all monitoring stages and rebuilding the pipelines on open source staged orchestration foundations. Moving to a workflow that links threat reports to locally deployed open inspection models not only reduces inference bills, but also gives the team full ability to audit each query stage and ensure the safety of critical systems without external reliance.