Skip to content

Transferring machine-learning code between NVIDIA and Google chips: the XStack benchmark reveals the limits of programming agents

Share
Transferring machine-learning code between NVIDIA and Google chips: the XStack benchmark reveals the limits of programming agents

Listen to this article

Read by Anchor

Developing machine-learning models today faces a sharp ecosystem split that limits code mobility, as projects are largely confined to two distinct environments: the PyTorch stack, which relies on NVIDIA GPUs, and the JAX stack, designed for Google TPUs. This software-and-hardware division prompted researchers Linrong Tsai, Taiming Luo, Hawtche Chen and Guang Liu to propose a new benchmark called XStack, based on the principle that machine-learning code should run wherever the researcher chooses to execute it, not only where it was originally written.

Moving code repositories between the two stacks is a practical necessity for decoupling from specific hardware and offering developers broader options, but it poses a complex engineering challenge because it involves whole repositories and requires rebuilding and verification across different frameworks and chips over long execution pipelines. The new standard adopts a strict protocol for testing programming agents: each agent receives a repository written in PyTorch on a clean machine equipped with TPUs and must deliver a complete repository that works in the target JAX environment. An agent passes only if its outputs meet precise success criteria directly tied to the published results of the original repository after verification and fine-tuned calibration.

The test results showed thatTransferring code repositories via agents remains unreliableto a degree that does not allow reliance on them without strict human review. Claude Opus 5 topped the performance list with a success rate of 75 percent, while GPT-5.6-sol fell to 45 percent, and the remaining tested smart agents failed to exceed the 40 percent threshold. This gap reveals that comprehensive programming tasks spanning long timelines and requiring the disentanglement of computational dependencies across hardware are still resistant to full autonomy, even for the most powerful models available on the market.

In our follow-up on infrastructure choices, this assessment carries clear strategic implications for engineering teams and computing research centers in the Gulf, Egypt and the Levant, where tech firms aim to diversify cloud-computing options and avoid full reliance on NVIDIA alone. Shifting to Google TPUs or the JAX environment opens a path to lower cloud-rental costs, yet attempting to use programming agents to automatically rewrite and migrate code introduces hidden costs in terms of risks and bugs. The results confirm that your technical team cannot yet delegate repository migration to agents independently; inspection, comparison and calibration of reference results must remain at the core of developers' work, lest the effort to save on hardware end in model-accuracy loss or training-pipeline failures.

The release of the XStack standard provides the first objective metric that measures real progress toward code transfer that crosses framework and hardware boundaries without obstacles. The practical message to technology leaders is thatFreeing code from hardware constraintsremains a legitimate engineering goal, but the path to it involves standard tests and precise calibration that agents cannot resolve on their own at present.

Don't miss the next story

Subscribe for updates