Skip to content

Training agents with audio and visual to run computers: a shift testing automation projects in the region

Share
Training agents with audio and visual to run computers: a shift testing automation projects in the region

Listen to this article

Read by Anchor

A research team that includes Guangchi Sun, Zhonghao Chai, Yishuan Li and Zhao Chang presented a new software framework called “VEX” that reshapes how AI agents are trained to use computer interfaces and operating systems through direct audio-visual understanding. We observed a direct impact of this technical shift on automation engineering and software development teams in the Gulf, Egypt and the region, as it narrows the historic gap between watching recorded tutorials and precisely executing tasks without having to manually split video clips into still frames.

Current multimodal large-language-model systems enable agents to perceive graphical user interfaces, interpret commands and carry out digital actions. However, the methods used today to train agents on video-based demonstrations are limited to converting clips into action sequences built from fragmented screenshots, which wastes the rich auditory and visual information that accompanies the explanation, such as voice-over narration and the live timing of on-screen movements.

The VEX framework introduces a three-stage workflow that researchers call “watch-execute-re-watch”. In the first stage, the intelligent agent extracts precise procedural instructions from educational videos that combine visual illustration with human speech. In the second stage, the agent carries out the required programming tasks on the computer interface. In the third stage, the system verifies the correctness and progress of its performance by re-watching the video of its actual execution to self-evaluate its behavior and review its trajectory.

Merging audio with interface motion eliminates the need for manual de-segmentation and gives agents the ability to self-verify their screens.To apply this framework in practice, the team built a multimodal computing agent named “video-SALMONN-VEX”, a nine-billion-parameter model based on the “video-SALMONN 2” architecture and supported by the base model “Qwen3.5-9B”. The team also devised a reinforcement-learning algorithm called “AGGO” to improve the instructions extracted from videos and link them to predictions of the next executable actions.

To support this expanded training, the researchers enriched the benchmark platforms AgentNet, OSWorld and CUA-Gym with more than 24,000 real-world instructional videos. Practical tests showed a clear and consistent advantage, as the video-SALMONN-VEX model achieved a relative 22 percent increase in task-completion success rate on the OSWorld platform compared with its base model Qwen3.5-9B. We also examined the efficiency of the guidance produced by the framework, finding that those instructions raised the success rate of the commercial model Gemini-3.5-flash by 10 percent on the same benchmark.

What this means for the region

This research puts digital-transformation teams and automation developers in regional organizations in front of a practical change in how operational procedures are digitized. Instead of draining budgets on writing complex textual instructions for each internal system or breaking employee recordings into thousands of still images, organizations can invest in video training clips with accompanying narration as they are, enabling agents to autonomously grasp complex tasks.

Achieving a nine-billion-parameter model built on an open architecture with a 22 percent success jump also opens a decisive economic option for companies in the Gulf, Levant and Egypt. This achievement allows the deployment of high-efficiency computer-operating agents on low-cost local servers, without relying exclusively on expensive closed-source cloud-call models, thereby protecting operational data privacy and strengthening technical sovereignty for sensitive work environments.

Don't miss the next story

Subscribe for updates