CapMem benchmark turns visual recordings into textual memory: temporal captioning outperforms direct video processing

Wearable smart assistants face a complex engineering dilemma when it comes to building a long-term episodic memory from first-person video recordings. Although operational needs require retrieving past events and understanding their details accurately, contemporary vision-and-language models run into concrete constraints imposed by limited frame-budget capacities, sharply rising costs of visual token processing, and retrieval failures over extended temporal contexts. To address these obstacles, a study led by Dingli Liang’s research team for the EMNLP 2026 conference introduced a benchmark called “CapMem”, aiming to demonstrate thatconverting visual scenes into textual captionscan constitute a renewable, reusable episodic memory that outperforms direct processing of raw video streams

The paper proposes a new task that involves answering questions based on episodic-memory captions, where the team built the CapMem benchmark using 75 video clips with full human review and annotation, totaling 33.7 hours, and comprising 1,000 multiple-choice questions spread across 16 realistic scenarios. The approach segments recordings into fixed temporal windows of 30 to 60 seconds to generate a sequential textual record of events, shifting the processing burden from intensive video-frame consumption to the lighter, more indexable and retrievable textual domain.

Laboratory results validated this approach, especially for videos longer than 20 minutes; the question system based on full captions with a 30-second window outperformed direct video processing in ten out of twelve tested models, and eight out of twelve models performed better when using a 60-second window. In a tightly controlled experimental setting ensuring frame parity across six models from the “Quinn” family, the textual approach retained accuracy gains averaging 3.22 and 2.55 points, respectively. Moreover, the experiments showed that usingthe text-guided retrieval and verification mechanismadded an additional improvement in answer accuracy of up to 5.3 points, confirming that selectively recalling visual frames after narrowing the search via text achieves the required efficiency without wasting computational resources

This conclusion is especially relevant for engineers and smart-system development teams in the Gulf, Egypt and the Levant, particularly for entities exploring the deployment of smart glasses and wearable cameras in field-inspection, supply-chain warehousing, and industrial-energy facility management. The continuous flow of video streaming hours into cloud inference models imposes costly visual-token consumption bills and strains network capacity, in addition to constraints on transmitting sensitive data. The new benchmark points to the feasibility of an alternative local architecture that offloads scenes into synchronized descriptive texts via lightweight edge processing, limiting heavy visual analysis to targeted moments only, thereby lowering the operating cost of field assistants and making large-scale deployment a commercially viable option.