Skip to content

Cleaning agents’ logs reduces task success: a study reveals the trap of hiding software bugs

Share
Cleaning agents’ logs reduces task success: a study reveals the trap of hiding software bugs

Listen to this article

Read by Anchor

Cleaning the agents’ context and stripping their logs of stumbling steps appears to be an obvious engineering decision to reduce token consumption, yet a new paper by six computer science and artificial intelligence researchers shows that this procedure leads to entirely opposite results. A log that looks polished while the external reality retains the error’s traces not only confuses the agent but also drives it to commit compounded mistakes that exceed what leaving the raw log untouched would cause.

The team comprising Igor Bakhoomov, Anurag Kol, Hongjun Liu, Bo Bang, Shafiq Guti, and Eric Naikamp posed a fundamental question: does the corrected content itself help the agent or hurt it depending on how it is phrased in the log and how it matches what the external world holds? To answer that, the researchers froze the agent midway through its task while precisely setting its real-world state, then performed experimental edits to the log or to the external environment before handing the task to a receiving model to complete, using two specialized test environments, AgentCE-Bench and AppWorld.

Results on the AgentCE-Bench platform showed a sharp contrast between two ways of correcting the agent’s error; when the agent made a mistake in entering a particular value while the operating environment remained unchanged,adding a truthful correction note alongside the recorded errorraised the task recovery rate and the receiving model’s success in completing the remaining goals from 51 % to 66 %. In contrast, when the log was cleaned by replacing the error with a completed action implying that the agent had performed the act successfully, the recovery rate fell to 44 %, lower than leaving the erroneous log as is, which achieved 50 %, proving that having the correct value in context does not guarantee the outcome if the log lies about the actual execution state.

This gap widened further in AppWorld environment tests, where the experiments tracked database writes and the impact of success and failure notifications sent to the agent compared to what actually happened to the data. When the log showed a success notification for a write that did not occur in reality, the overall task success dropped from 45 % to 28 %. Conversely, a truthful success notification raised the completion rate from 38 % to 49 % when an actual change occurred in the database. The performance gap between the two cases reached 27 percentage points when aggregating results across the two receiving models, with this variance confirmed for seven of the eight receiving models tested by the researchers at a 95 % statistical confidence level.

These figures put system architecture in front of a practical reality that touches the technical gap that builds independent agent platforms in banks and logistics companies in the Gulf and Egypt; they drive the cost of model invocation for engineers to rely on memory-compression mechanisms and dropping stalled logs to cut operating expenses. The research data show thathiding transaction failures misleads subsequent modelsand causes them to base their decisions on the assumption of update completion, leading to cascading process failures and repeated erroneous queries. The engineering skill required is to move from a superficial cleaning culture of context to engineering explicit observations that acknowledge error and document failure accurately, rather than erasing it programmatically.

The experiment reveals that models in real-world work environments require full alignment between log language and the surrounding physical data reality. Attempting to beautify the log to reduce tokens or claiming task completion yields counterproductive and costly results: the agent learns to handle and correct its mistakes when they are clearly logged, whereas its performance collapses entirely when it is told the operation succeeded while reality still retains failure traces.

Don't miss the next story

Subscribe for updates