Running trillion-parameter models on 59 giant chips breaks memory bottleneck in inference servers

AI models with capacities of trillions of parameters pose a complex hardware challenge that exceeds the capabilities of individual chips, as their weights alone require storage measured in terabytes within fast memory. Cerebras, a US AI hardware company, disclosed architectural details for running advanced multi-trillion-parameter models on its giant WSE-3 chips, using the open-source Kimi K3 model with three trillion parameters as a practical demonstration of achieving inference speeds of between one thousand and two thousand tokens per second.

Test data showed that the model, which comprises 69 linear attention layers and 24 dense layers for a total of 93 layers, delivers performance comparable to GPT-5.6 Sol, a proprietary model, on agent and programming tasks, scoring 88.3 percent on the terminal-environment benchmark versus 88.8 percent for the counterpart model, and 91.2 percent on web browsing, surpassing 90.4 percent. To keep the system stable on the chips, the architecture quantized the model’s weights to 4 bits, limiting each layer to roughly 16 gigabytes, while using 8 bits for key-value memory and assuming an operational context of 256 thousand tokens per request.

Storing the model’s weights in the integrated static memory eliminates the need for continuous transfer over the network bandwidth.Because each full silicon chip contains 44 gigabytes of on-chip static RAM, the software compiler distributed the layers according to their operational characteristics. Dense layers require additional key-value memory per token for each active request, which led to allocating an entire chip for each dense layer, totaling 24 chips, while leaving the remaining memory space for context storage. Linear attention layers maintain a small fixed state per request regardless of context length, allowing two linear layers to be placed on a single chip, resulting in 35 chips, bringing the total system to 59 interconnected chips.

This distribution operates through a sequential pipeline where the chips perform synchronized processing similar to industrial assembly lines. Model weights never move between chips during operation; only intermediate activations are passed over an Ethernet network that supports remote direct memory access, with a transfer latency of about two microseconds per hop between chips. The architecture allows multiple small batches of requests to flow through the processing line simultaneously without waiting for one request to finish before starting the next, as each chip handles a different stage of an independent request, making the maximum number of concurrent requests limited by the remaining key-value memory in the dense-layer chips.

The true efficiency metric for advanced inference systems is measured by token throughput per unit of energy consumed, not merely by batch size.Our engineering analysis shows that this shift touches the core technical decisions of infrastructure teams and data centers in the Gulf and Egypt, where banks and cloud providers face high costs and energy and cooling bottlenecks when attempting to run large open-source models on traditional GPU arrays. When system engineers and independent agents in the region need to reduce response time for complex reasoning chains, architectures that transfer only activations and embed trillions of parameters in fast memory enable locally sovereign models to run with superior performance without draining bandwidth networks.

This experiment demonstrates that scaling ultra-large models no longer depends on waiting for larger single-chip capacities, but on the software-hardware system’s ability to evenly partition model layers and link chips within a connected processing pipeline. With the possibility of adding extra chips to support models exceeding three trillion parameters, the infrastructure race is moving toward reducing data movement between processors and relying on ultra-close memory to ensure stable, continuous inference.