Task Model Induction research framework extracts structured workflows from computer usage traces

A research team comprising Yucheng Jiang, Zora Zhiruo Wang, Ruixi Chen, and Diyi Yang introduces a new framework called Task Model Induction, aimed at extracting symbolic, auditable, and reusable models of daily workflows directly from computer usage traces. The methodology relies on automatically recorded data, including sequential screenshots, mouse movements, and keyboard inputs, to convert these raw operational behaviors into actionable knowledge structures.

These models gain critical importance as computer-use agents begin deploying in real work environments, where intelligent systems must understand how tasks are actually executed, while organizations require methods that allow them toreview and repurpose operational knowledgesafely. Even so, task model induction faces fundamental obstacles in real-world application, as activities in operating environments are recorded merely as low-level events, whereas practical work is multi-track and tangled across concurrent execution goals.

The paper, published on arXiv, notes that prior methods suffered from clear limitations because they assumed a predefined task or a simple single-track workflow, restricting their outputs to brief step-level summaries rather than complete structural task models. In contrast, the new framework operates in two main stages, beginning with discovering latent tasks within unconstrained traces and untangling concurrent activities carried out simultaneously by the user.

The second stage of the framework builds a dedicated model for each discovered latent task bycombining a hierarchical goal model with a procedural control-flow model. The hierarchical model iteratively breaks down overall goals into subgoals, while the procedural model organizes and tracks the control flow driving actual execution steps, allowing a shift from merely tracking individual clicks and inputs to a comprehensive structural understanding of workflow steps and branches.

In empirical evaluations on controlled trajectories from both human and computer-use agent interactions, the proposed framework recovered interleaved tasks with an alignment score of 0.974 against ground truth annotations. It also successfully reconstructed 74.9 percent of observed execution steps, outperforming the strongest workflow induction baselines in the research literature by a clear margin.

When testing induced skills on held-out tasks, models derived through the new framework showed a 30.0 percent improvement in performance accuracy compared to the strongest baseline, underscoring the effectiveness of converting screen and input traces into precise procedural rules that help onboard agents and ensure process transparency across organizations.