Heterogeneous Compute for Democratizing Access to AI: From Workload Awareness to Scalable Deployment

Abstract

The panel examined how moving beyond a GPU‑only compute model toward a heterogeneous mix of CPUs, NPUs, TPUs and GPUs can make AI more affordable and globally reachable. Topics covered ranged from disaggregated, network‑embedded compute and edge‑cloud orchestration to the three major impediments to AI adoption in India—power & infrastructure, security/safety, and data gaps. Participants highlighted practical deployment examples (education, SMBs, sovereign LLMs), environmental and cooling considerations, and policy‑level levers (energy, water, land) required to enable a trustworthy, scalable AI ecosystem.

Detailed Summary

• Bhawna Agarwal (HPE) opened the discussion by stressing the need for voice‑first interfaces in native Indian languages. She argued that to deliver seamless user experiences, AI inference must be available both in the cloud and locally on devices, especially when network connectivity fluctuates.
• She highlighted recent hardware advances: a 10 billion‑parameter multimodal model can run on a modern smartphone, while sub‑1 billion‑parameter models can operate in smart glasses with only a once‑daily charge.
• The environmental angle was introduced: efficient compute reduces overall energy consumption, a critical point for a country with finite power resources.

2. The “Three Impediments” to AI Adoption (Cisco Perspective)

• Speaker: Daisy Chittilapilly (Cisco) (referred to as “Arun Shetty” in the transcript) summarized the impediments identified by the panel:

  1. Infrastructure Constraints – Power demand projected to reach 63 GW in the next few years (citing U.S. power forecasts). Compute capacity is also tightening, and networking bandwidth will become a bottleneck.
  2. Security & Safety – Trust is foundational: “you can’t trust what you can’t see.” Concerns include model hallucinations, malicious data injection, and the need for visibility across the entire AI stack.
  3. Data Gap – High‑quality, domain‑specific data is the fuel for AI; without it, building effective models is impossible.

• Cisco’s stance: develop fit‑for‑purpose solutions that combine centralized data‑center capability with edge inference, reducing latency and bandwidth consumption.

3. Sovereign Models & National Resilience (Prof. Kamakoti & Minister)

• Prof. Kamakoti V. (IIT Madras) emphasized that critical public infrastructure must be powered by sovereign AI models to mitigate adversarial attacks and data sovereignty risks.
• He introduced a trust framework grounded in discrete mathematics (reflexivity, symmetry, transitivity) to formalize context‑dependent trust among AI components.
• Shri D. Sridhar Babu (Telangana Government) reinforced the policy perspective, underscoring that AI rollout must respect power, water, and land constraints. He positioned the government as a facilitator that will provide the essential utilities while technologists supply the compute solutions.

4. Heterogeneous Compute in Practice (Qualcomm & Intel Views)

4.1 Qualcomm – “Hybrid AI”

• Dr. Durga Malladi (Qualcomm) described Qualcomm’s Hybrid AI strategy: a seamless blend of device‑level inference, edge‑cloud, and large‑scale data‑center processing.

• Key points:

  • Edge devices handle low‑latency, low‑power inference; cloud back‑ends manage heavy training and large‑parameter models.
  • Air‑cooled servers can comfortably support 100‑300 billion‑parameter models at the edge‑cloud tier, limiting the need for expensive liquid‑cooling.

4.2 Intel – Energy‑Efficient Deployment

• Gokul Subramaniam (Intel India) focused on vertical‑specific workloads and the hardware constraints that affect edge AI: memory bandwidth, I/O, thermal limits, and power budgets.

• He highlighted three sectors:

  1. Education – Real‑time translation and transcription on low‑power devices to broaden access.
  2. Small & Medium Business (SMB) – Deploying edge inference to enable affordable AI‑driven tools.
  3. Power & Cooling – In data‑centers, 40 % of energy goes to cooling, 40 % to compute, 20 % to networking. Improving Power Usage Effectiveness (PUE) toward 1.0 is essential.

• Intel is exploring hybrid energy solutions (mix of renewable and conventional power) and liquid‑cooling technologies for racks exceeding 25 kW, to keep Indian data‑centers sustainable.

5. Security, Safety, and Governance

• Security Concerns (Cisco & Intel):

  • Dynamic malware signatures now require AI‑powered inspection that can adapt in real time.
  • Shadow AI (unaudited AI applications) creates blind spots; organizations must discover, inventory, and continuously scan AI assets.

• Safety Measures (Qualcomm & Intel):

  • Implement guardrails that block the transmission of confidential data to external LLMs.
  • Use NIST, MITRE, OVAS‑P guidelines to benchmark model robustness.

• Policy Recommendations (Minister & The Dialogue):

  • Build sovereign model repositories under governmental oversight.
  • Align AI deployment with national energy‑security policies, ensuring that AI does not exacerbate power shortages.

6. Real‑World Deployment Examples

Use‑case	Hardware Mix Recommended	Key Benefits
Language translation for rural schools	Low‑power NPU‑enabled tablets + edge‑cloud micro‑data‑centers	Real‑time translation, minimal bandwidth, low energy
SMB predictive maintenance	CPU‑GPU hybrid on‑prem servers with edge sensors	Cost‑effective inference, reduced latency
Smart glasses for field workers	Sub‑1 B‑parameter model on NPU‑rich wearables	Battery life >24 h, offline capability
National health data analytics	Secure, sovereign LLMs hosted in government‑controlled data‑centers, with edge pre‑processing	Data sovereignty, compliance with privacy norms

7. Audience Q&A (Key Questions & Answers)

Question	Speaker (Answer)	Summary
How can we ensure AI models remain trustworthy when they are distributed across devices?	Prof. Kamakoti	Trust must be defined mathematically; models need context‑aware trust policies and continuous validation.
What concrete steps should enterprises take to discover shadow AI?	Daisy Chittilapilly (Cisco)	Deploy AI asset‑discovery tools, enforce inventory policies, and apply automated vulnerability scans.
Is liquid cooling mandatory for Indian data‑centers?	Gokul Subramaniam	Not yet; air‑cooled solutions are viable up to ~25 kW per rack. Liquid cooling becomes necessary beyond that, but hybrid cooling strategies can lower overall PUE.
How can policy accelerate sovereign model development?	Shri D. Sridhar Babu	Government should provide funding, data‑set access, and regulatory sandboxes to accelerate local model training.

8. Closing Remarks

• Minister Shri D. Sridhar Babu reiterated that AI must serve welfare and happiness for all, stressing the need for coordinated action on power, water, land, and policy to enable equitable AI access.
• Kazim Rizvi (The Dialogue) thanked the panel and highlighted that the dialogue itself is a step toward inclusive AI policy formation.

Key Takeaways

• Heterogeneous compute (mix of CPUs, GPUs, NPUs, TPUs) is essential to balance cost, performance, and energy efficiency for AI at scale.
• Three primary hurdles in India: (1) Infrastructure (power, compute, networking), (2) Security & safety (model trust, adversarial threats), (3) Data gaps (need for high‑quality, domain‑specific datasets).
• Edge inference is becoming a dominant trend; devices can now run 10 B‑parameter models, dramatically reducing reliance on continuous cloud connectivity.
• Fit‑for‑purpose solutions—tailoring hardware to workload characteristics—are more sustainable than monolithic GPU‑only data‑centers.
• Trust frameworks grounded in formal mathematics are proposed to define and enforce AI trustworthiness across heterogeneous stacks.
• Energy & cooling: 40 % of data‑center energy is consumed by cooling; improving PUE and adopting air‑cooling up to 25 kW per rack can curb power waste. Liquid cooling should be reserved for higher‑density deployments.
• Security governance must include AI asset discovery, continuous scanning, and guardrails that prevent leakage of confidential data.
• Policy action: Government must ensure stable power, water, and land supply, foster sovereign model ecosystems, and create regulatory sandboxes for safe AI experimentation.
• Real‑world pilots in education, SMBs, and smart wearables demonstrate the viability of heterogeneous, edge‑centric AI deployments.
• Collaboration is vital—industry, academia, and government must co‑design the AI stack to achieve democratized, trustworthy, and energy‑efficient AI across India.

---

See Also:

• democratizing-ai-resources-in-india
• ai-innovators-exchange-accelerating-innovation-through-startup-and-industry-synergy
• a-billion-voices-one-ai-how-language-tech-transforms-nations