EpiSEM-Bench test: AI agents execute prohibited tasks when tools hide safety facts

Engineering experiments in the smart agent pipeline reveal that the greatest security risk does not necessarily stem from malicious user requests, but seeps from a precise knowledge gap: the model receives completely accurate and truthful data from its tools, yet overlooks a single crucial fact tied to safety constraints. In this case, the agent proceeds to execute a prohibited action in full compliance, even though the user’s request was innocent and contained no intent to deceive.

This challenge prompted a research team that includes Kijin Chang, Hantshen Wang, Yuhuan Chao, Haoyu Dai, Yao Shu, Zhongxiang Dai, and Shuang Qiu to develop an applied evaluation benchmark they named “EpiSEM-Bench”. The benchmark targets the examination of unsafe execution rates, the exploration of safety boundaries, and the testing of models’ ability to retrieve missing evidence when their tools omit essential facts, using 300 matched pairs of safe and unsafe tasks across six different domains.

A language model may slip into a grave error simply because the tool did not explicitly prohibit it, even if every piece of information it provided was factually accurate.By systematically and deliberately withholding critical facts, the benchmark measures prohibited execution decisions caused by omitted evidence and determines the minimum observed failure across various levels, ranging from failures that occur despite complete evidence to cases where no failure appears after exhaustive test rounds.

The researchers relied on an independent auditing tool to monitor whether the agent actively seeks missing facts and refrains from prohibited actions while continuing to complete authorized tasks. Testing ten language models revealed a wide variation in their robustness to deleted facts, evidence-gathering mechanisms, and appropriate decision-making; it became clear that a model’s immunity to incomplete data does not imply its ability to retrieve and use that evidence effectively.

While granting models the ability to retrieve evidence reduced unsafe execution rates across all ten models, the differences remained evident in how proactively the models searched for missing facts and used them to act correctly on both tracks, whether the task was safe or required a halt.

This conclusion reframes technical safety requirements for AI engineers and CTOs in the Gulf, Egypt, and the Levant; limiting protection to cleansing command texts does not provide a robust barrier for sensitive operations.When smart agents connect via APIs to banking databases in Riyadh and Abu Dhabi, or to logistics and enterprise service platforms in Cairo and Oman, the danger lies in the system’s truthful yet incomplete response. If the data interface confirms the client’s identity or account balance but omits the regulatory comment field or an ongoing audit, the agent will proceed with the transaction believing its path is correct.

Today’s practical duty for development teams in the region goes beyond monitoring models’ final outputs, toward building proactive verification layers and independent auditing tools that compel the agent to confirm that safety context is complete before granting it decision-making authority over the underlying business infrastructure.