Ziroh Labs, the homegrown AI startup, has developed a new computing platform that claims to reduce reliance on costly and scarce GPUs for running large AI models. The system, called Kompact AI, has been built in collaboration with the Indian Institute of Technology (IIT) Madras, and is intended to allow advanced AI systems to operate on CPUs found in standard computers.
Ziroh Labs’ Kompact AI is designed specifically to focus on AI inference — the process of using trained AI models to perform tasks like answering queries or generating content. While GPUs are still critical in training large models due to their ability to process vast amounts of data simultaneously, Kompact AI aims to run these models after they have been trained, using the far more commonly available CPUs. This approach was demonstrated at an event held recently, where Ziroh Labs showcased AI models like Meta’s Llama 2 and Alibaba’s Qwen2.5 operating on a laptop powered by an off-the-shelf Intel Xeon processor. According to the company, the system could process queries effectively without specialized hardware.
Kompact AI is part of a larger effort undertaken by the Centre of AI Research (COAIR), jointly established by IIT Madras and Ziroh Labs. The Centre focuses on developing AI systems that are accessible, affordable, and suited for real-world use in regions with limited computing resources. This also aligns with India’s broader national vision of “AI for All,” which seeks to foster inclusive AI development that can operate in diverse environments without the heavy infrastructure requirements seen in the West.
This development comes at a time when access to advanced AI infrastructure has become a key concern for developers, particularly in countries like India, where resources for large-scale computing are limited. The global demand for GPUs has pushed up costs and made them difficult to acquire, even for large companies. IIT Madras Director V. Kamakoti commented on this, saying, “The AI divide is because only those with high-end expensive GPU-powered resources can access, develop and deploy powerful AI,” he said. “We’re demonstrating that you don’t need a revolver to kill a mosquito.” Thus, it seems that the most immediate impact of Kompact AI is its potential to level the playing field in AI development, ensuring that smaller startups, educational institutions, and researchers in developing countries cease to struggle to meaningfully contribute to the AI landscape by enabling large AI models to run efficiently on standard CPUs (thus lowering the barrier to entry for organizations without deep pockets).
Ziroh Labs stated that its technology has been tested by major US chip manufacturers, including Intel and AMD, to validate its performance. The platform has reportedly been optimized to handle 17 widely used AI models on CPU-based systems. According to the startup, models with parameters under 50 billion — typically used in various business and research settings — perform well on Kompact AI without compromising accuracy. William Raduchel, a former executive at Sun Microsystems and an advisor to Ziroh Labs, suggested that it will have a very profound market impact in the years ahead.”
