Togider launches continuous inference translator: agent loop breaks monopoly of manual tuning of compute kernels
Listen to this article
Read by Anchor
The pace of launching AI models is accelerating and their architectures are diversifying beyond engineers’ ability to write and tune kernels manually.This acceleration places the computing infrastructure operators in a perpetual technical dilemma; each new architecture and variable traffic distribution requires rewriting and precisely tuning computing paths to ensure efficient and sustainable operational performance.
Researchers at Togider AI presented a paper in the Archive of Arkaiv titled the Togider Translator system, known as TKCC, as the first continuous software agent loop entirely designed to improve AI inference. The system relies on gradient-climbing algorithms to execute incremental, continuous optimisations of the forward-pass path, bypassing the need for slow human intervention in re-tuning and re-assembly.
Previous attempts at continuous compilation collided with a complex trade-off between measurement speed and precision. Systems that use agents to optimise single computational kernels through isolated tests outside the real production line provide low-precision measurements because of data-driven behaviour in vital computational structures such as Mixture-of-Experts (MoE) models. Conversely, systems that employ agents to optimise inference engines directly suffer from long wait times and very slow launch of software engines, hindering the repetition of experiment-and-test cycles.
TKCC overcomes the measurement and time-waste dilemma through an innovative two-part architecture:The first part is a multi-layer continuous engine that separates the software engine launch burden from forward-pass calculations, while the second part is an analytical cost model that selects the optimal parallel configurations from which automatic optimisation algorithms for inference agents launch.
Experimental results show that this translator outperforms meticulously hand-written megakernels by up to 1.7 times on the widely deployed Llama 1B model. It also outperforms leading open-source engines when running the latest advanced models, including ChemK3, GLM 5.3, MiniMax M3.1 Flash, and DeepSeek V4.1 Flash, achieving inference speed up to 2.6 times higher in tokens per second on the Inference X benchmark across multiple hardware platforms.
The translator set unprecedented records when handling single-inference requests with a batch size of one, recording 1,111 tokens per second on the M3.1 Flash model and 1,205 tokens per second on the DeepSeek V4.1 Flash model, while also entering real-time operation for a trillion-token service on the company’s production servers.
This achievement has a direct economic and organisational impact on data centres and engineering teams in the Gulf, Egypt, and the Levant. Regional institutions in the financial and governmental sectors rely on locally deploying open-weight models within private cloud environments to ensure digital sovereignty, yet they bear high computing costs due to inefficient virtual kernels with complex MoE architectures, and they suffer from a shortage of developers specialised in low-level kernel tuning.
Lowering inference cost shifts from a rare engineering skill to an automatic software layer that continuously operates in the system background.This shift allows regional development teams to reduce the response time of interactive smart systems and real-time decision-making aids, and to lower computing hardware consumption in production environments, without waiting for model-development companies to release updates or hiring additional teams to rewrite computational code manually.