How IIT Madras, Ziroh Labs Are Ditching Expensive GPUs in Favour of CPUs - Explained

While India is going an extra mile to become a $1 trillion digital economy by 2030 amid the global tech race, the country’s AI dreams are still heavily dependent on foreign technology. Despite being home to 16% of the world’s AI talent, Indian start-ups and researchers face significant hurdles in AI innovation, including the absence of homegrown large-scale artificial intelligence models, and the high cost of hardware infrastructure.

However, Union Minister Ashwini Vaishnaw had earlier informed that the IndiaAI mission that was started one year ago, has made 15,000 GPUs available now. Of these, top chipmakers like Nvidia, AMD, Intel and Amazon Web Services have provided a substantial number of GPUs to India.

Currently, India is also in the process of making its own GPUs, which are expected to be ready in the next 3 to 5 years. But this raises an important question: Is relying solely on GPUs the only way to run AI models, especially for a country like India which is far behind from China and the US in AI race?

The answer is no! Researchers at IIT Madras, along with India’s home-grown deeptech start-up ‘Ziroh Labs’ have come out with an ‘affordable’ alternative. They have designed an AI system called 'Kompact' that can run large artificial intelligence models without advanced computing chips from the likes of Nvidia Corp.

But before diving deep into the platform, let's understand the concept of GPUs and CPUs first. Generally, GPUs are preferred over CPUs for their superior parallel processing capabilities, which are crucial for deep learning. While CPUs excel at sequential processing and general-purpose tasks, GPUs are designed to handle massive parallel operations.

What Is Kompact AI?

The team has launched ‘Kompact’, an AI platform that could tackle India’s compute accessibility gap by eliminating the country’s reliance on costly GPUs. The platform, on the contrary, enables foundational models to be inferences using CPUs (central processing units).

“Training large models is incredibly expensive,” said Prof. Madhusudhanan B, Principal Consultant - IITM Pravartak Technologies Foundation, which also joined hands with IIT Madras and Ziroh Labs in the creation of Kompact AI. Citing an example of OpenAI, he said the ChatGPT maker reportedly spent anywhere between $75million to $100 million on training its chatgpt4.

The professor further explained the practical applications of Kompact AI, highlighting how the platform aims to democratise artificial intelligence by enabling powerful AI models to run efficiently on CPUs.

How Does Kompact AI Work?

With an aim to enhance accessibility, the Kompact enables inferencing --- the process of a trained AI model making predictions or solving tasks using new or unseen data --- for models with less than 50 billion parameters. It outperforms existing CPU-based systems by nearly three times.

According to him, inferencing is important because it’s the phase where 99.99% of users actually engage with the model after it has been trained. However, a major challenge with inferencing today is the GPU shortage, which is typically required to run these large models efficiently. And a person even needs to wait for six months to get hands on a good GPU.

That’s where the idea of using CPUs came to Ziroh Labs and it started working on it. “They went deep into the transformer architecture, understood the science and math behind it, and rewrote the algorithms to be optimised for CPUs. These proprietary algorithms are Ziroh’s breakthrough,” he explained.

"At IIT Madras, we benchmarked Ziroh’s tech with open-source models like LLaMa CPP and found a 3x speed improvement on CPU-based inferencing,” Prof. Madhusudhanan B added. In short, Kompact’s new approach retains the full capabilities of the AI model while enabling efficient CPU-based deployment. However, traditional methods like distillation or quantisation often compromise performance and accuracy to make such models CPU-compatible.

The professor also claimed that at least 20 start-ups from the IIT Madras research and start-up ecosystem have either shown interest or started integrating Ziroh’s Compact AI into their stack. The start-up, as quoted by Bloomberg, claimed that its technology has been tested by US chipmakers Intel and Advanced Micro Devices.

Till now, Ziroh Labs has optimised 17 AI models, including DeepSeek, Llama, Qwen, BERT, Phi, etc., to operate them efficiently on CPUs. Currently, it is working on various other models like Gemma, Moonshine, GoogleNet, OpenPose, ShuffleNet, etc.

How Will It Help Start-Ups?

Start-ups can leverage the platform to run their AI use cases on the cloud, on-premise or even on laptops. It opens up use cases in finance, law, healthcare, manufacturing — anywhere data privacy is critical. With Kompact AI, these organisations don’t need to send data to the cloud or rely on GPU-heavy setups.

Besides this, this technology has clear cost benefits. While exact numbers vary, a 50% reduction in inferencing costs is a fair baseline assumption when comparing CPU vs GPU deployments, the professor said.

As per The Economic Times report, the platform aims to onboard at least 200 startups over the next two quarters—offering free access for up to a year if needed—to validate its capabilities and support a broad range of use cases

“Ziroh Labs is focused on enabling start-ups and academia through translational research, and we at IIT Madras are proud to support and validate their work,” he added.

While India is working on developing its own indigenous GPU expected to debut in next few years, alongside the deployment of 18,000 AI servers, this home-grown start-up fills a crucial gap in the short term. For now, GPUs may have fueled AI’s global ascent. But India’s future in artificial intelligence might well be built on something more sustainable.