SlackBench tests agents in shared development environments: most models under 66 percent
A recent research study, published by researchers including Mingxuan Li, Alex Demakis and Joseph Gonzales on the arXiv platform in early November 2026, revealed an operational shortfall faced by AI agents when tested in real collaborative work environments. The team presented a new evaluation framework called “SlackBench”, designed specifically to measure software models' ability to perform joint inference across team conversation logs and continuously evolving code repositories.
Previous efficiency tests usually measured model capabilities using synthetic data or isolated programming tasks, which painted an overly optimistic picture of agents' ability to act as independent digital colleagues. However, field work imposes a different reality, as essential data are distributed across public communication channels, instant messages, repository changes and decision updates. In such environments, answering a simple technical query is not enough by reading an isolated file; it requires identifying the relevant discussion channels, tracking long conversation threads, examining repository state, and reconciling conflicting or outdated evidence.
The engineering environment requires linking the evolving discussion to the code state.
The SlackBench benchmark was built on realistic, privacy-preserving simulations of academic research projects, providing reproducible execution environments that combine developer discussions and code changes via the GitHub platform and Google Docs documents. The benchmark contains 140 precise queries covering four functional areas: intelligent search across heterogeneous sources; justified refusal to answer when impossible questions or those based on false assumptions arise; summarization directed toward specific goals; and inference of contexts and conditions from long conversations.
Evaluation results covering 33 leading models and an agent-execution suite showed a clear gap in field performance; the best system did not exceed the 80 percent threshold, while most agents settled below 66 percent. This gap indicates that collaborative tasks cannot be reduced to answering static questions, as success demands advanced ability to audit change logs and sift dialogues to extract the correct context.
The lack of ability to refuse answering amplifies silent errors.
This digital divide reshapes the calculations of technology managers and software team leaders in the Gulf, Egypt and the Levant, amid the accelerating push to connect smart agents with enterprise communication platforms and code repositories for documentation management and assisting new engineers. If the top systems err in one-fifth of cases and most agents fail in more than a third of tasks, direct reliance on these tools for architectural decisions or software status tracking threatens the integration of faulty changes based on cancelled discussions within work channels.
This shift forces regional developers to revisit publishing strategies used within organizations. Value is no longer limited to issuing commands or invoking advanced models; it now requires engineering strict verification frameworks that compel the agent to stay silent and acknowledge missing data instead of fabricating answers based on copied context, and to ensure its responses align with the latest operational state of the code repository.