VLoc benchmark reveals limits of vulnerability scanning, with agents collectively failing to locate flaw locations in 38 percent of repositories

Typical cybersecurity assessments of software agents based on language models focus on measuring the ability to detect, reproduce, or patch vulnerabilities, ignoring the foundational step that precedes all that: whether the agent can identify the actual source code files responsible for the flaw within an entire software repository. To answer this question, a research team including Aman Brianshu, Sobriety Vijai, and Amin Karbasi presented a study measuring agents' efficiency in locating vulnerabilities at the level of full code repositories, emphasizing that this task constitutes an independent capability requiring precise testing standards before relying on agents for complex system reviews.

The researchers introduced the evaluation benchmark called “VLoc Bench”, which includes 500 real security vulnerabilities extracted from 290 software repositories spanning six software package ecosystems and covering 147 CWE categories. The test design provided each agent with two versions of the code repository: one snapshot taken immediately before the security patch, and another taken immediately after. In the vulnerable version, the agent receives only the vulnerability classification description with read-only, limited repository access and must precisely extract the affected source files, while in the patched version it is asked to confirm that the recorded vulnerability has been removed and is absent from the code.

The test results revealed a wide operational gap in repository analysis across a unified agent interface for 27 language models and four static analysis tools.The top performance achieved by the strongest tested systems on the file F1 score halted at just 0.229, while all models and tools combined failed to provide a single correct identification of the vulnerability location in 38.4 percent of tasks. The study revealed a systematic flaw: models that show higher efficiency in detecting compromised files fail to handle code that has been patched, continuing to report locations not supported by facts within the patched repositories.

These results highlight that locating faults within a large code repository is an independent inspection skill that requires knowing when to stop and refrain from issuing false reports. The tendency of agents to assume that vulnerabilities persist even after they have been addressed doubles the workload of audit teams and prevents the full delegation of safety decisions to automated tools, which lack discipline in assessing the end of the software risk.

This test is linked to the reality of software engineering and cybersecurity teams in the Gulf, Egypt, and the Levant, where financial institutions and technology companies are moving toward integrating AI agents into code-audit pipelines to reduce costs and accelerate development pace. The study’s results for information security managers in the region show that delivering complete code repositories to agents without strict human supervision will cost development teams wasted hours chasing false reports, especially in banking and government systems subject to stringent compliance standards. The skill now required in local work environments is no longer just running a scanning agent, but building verification barriers that separate the agent’s detection capability from its ability to recognize the completion of security patching.