Bonsai 2 shrinks Qwen to 5.9 GB: triple-weight technique brings advanced inference to personal devices

The AI research lab PrismML launched its latest model, Bonsai 2 (27B), announcing a technical step that bets on shrinking the size of large language inference models without sacrificing their computational capabilities. The new release builds on compression of the open-source Qwen 3.8 (27B) model from Alibaba, reducing its total size to just 5.9 GB. This reduction cuts memory requirements by roughly nine to ten times compared with the original version, making the model capable of running directly on personal computers and possibly on advanced smartphones.

The experiment shows that advanced generative AI does not necessarily require massive servers, but can have its weights reduced to run on readily available desktop hardware.The lab was founded by a group of researchers from the California Institute of Technology (Caltech) and is led by Professor Babak Hosseini, a faculty member at the institute who specializes in computational compression techniques. Its advisory board includes Ion Stoica, co-founder of Databricks and director of the Sky Computing Lab at the University of California, Berkeley, who has launched notable projects such as Lita and SGEL-ENG. The lab is financially backed by Khosla Ventures, Cerebras Capital and Caltech after an initial funding round of $22.25 million, amid circulating reports of talks with Apple, a matter Hosseini declined to comment on in his press interview.

The technology developed by the company focuses on modifying the digital weights that store the knowledge acquired during model training. While conventional weights use 16 bits per weight, the company’s approach relies on ternary weights, which compress the stored values to only three possibilities: +1, 1, or 0. This sharp computational compression drastically reduces the storage and runtime footprint, placing the company in competition with other groups working on model compression, such as the Spanish firm Multiverse Computing, which originates from the Donostia International Physics Center.

The announced performance figures show that Bonsai 2 matches 98 percent of the overall results on the standard benchmarks for the original Qwen model, compared with the 95 percent achieved by the first release launched last March. The first version recorded more than 11 million downloads, while the company’s smaller models added roughly 2.6 million additional downloads. Hosseini believes that striving for a perfect 100 percent match remains a theoretical issue, as the uncompressed original models already exhibit some inaccuracy, standard benchmarks do not fully reflect model performance on real-world tasks, and the surrounding software and containers in which the model runs play a decisive role in determining its overall accuracy.

The lab’s focus in the next phase will shift to applying compression techniques to models containing hundreds of billions of parameters.Hosseini explains that larger model sizes provide more room for weight compression without sacrificing the model’s inference capabilities, anticipating that higher match rates with massive models will become attainable within the coming few months. Advisor Ion Stoica argues that the real value of this path lies in making advanced AI freely available to end users by running it on devices they already own, while fully protecting data privacy because inputs need not be sent to cloud servers.

This engineering development opens a practical path for developers and tech firms in the Gulf, Egypt and the Levant, directly altering the calculus of computing costs and privacy requirements. In terms of cost, organizations and development teams in Riyadh, Dubai, Cairo and Amman can forgo recurring expenses for cloud API calls and instead run 27-billion-parameter inference models locally on available workstations. Regarding digital sovereignty and regulation, the reduced model size enables banks, health sectors and government agencies in the region to build advanced AI solutions without exporting sensitive transaction data to global cloud providers. The shift also creates new skill requirements for regional engineering teams, pushing them to master the design of local runtime environments and the tuning of model containers rather than relying solely on ready-made APIs.