India AI Impact Summit 2026

quality-control-of-ai-applications-in-healthcare

6 min read

TRACK:Safe and Trusted AI
SPEAKERS
  • Lt Col Parikshit Sanyal, AFMC Pune
  • Nisheeth Srivastava, IIT, Kanpur
  • Tapan K Gandhi, IIT Delhi
  • Taruna Madan Gupta, ICMR
  • Vivek Hande, DGAFMS
SESSION OVERVIEW

Deploy AI models in healthcare through phased pilots, using clinically validated datasets and bias audits. Establish strict data governance, patient consent, and privacy compliance. Implement human-in-the-loop oversight for all clinical decisions. Mandate regulatory approvals, model explainability, and documentation. Conduct continuous performance monitoring, post-deployment audits, and retraining. Define clear accountability, cybersecurity safeguards, and standardized quality benchmarks to ensure safety, accuracy, transparency, and trust in real-world clinical use.

VIDEO RECORDING

Quality Control of AI Applications in Healthcare

Abstract

The panel explored why rigorous quality‑control mechanisms are indispensable for AI tools that support clinical decision‑making. Panelists described the spectrum of quality risks (data bias, lack of transparency, cybersecurity, regulatory gaps) and outlined a lifecycle‑wide governance framework that includes data governance, human‑in‑the‑loop oversight, continuous post‑deployment monitoring, and clear accountability structures. The discussion also touched on practical implementation steps such as a national AI‑governance policy (to be released by the Health Minister), a mandatory “kill‑switch” definition, the need for local validation, and the balance between open‑source collaboration and regulated, version‑controlled releases. Audience questions highlighted clinicians’ concerns about trust, cognitive debt, and the importance of incremental, ethically‑guided deployment.

Detailed Summary

  • Moderator (Lt Col Parikshit Sanyal) welcomed the audience and introduced Dr Karthik Adapa (WHO Digital Health) as a key architect of India’s forthcoming national AI‑for‑health strategy.

  • He framed the discussion with three clinical‑lead questions for DG AFMS Vivek Hande:

    1. How to differentiate a high‑performing AI model from a safe one?
    2. What “blind‑spots” or edge‑cases are most common in AI‑driven care?
    3. Will AI eventually cover the full gamut of medical practice or only selected stages?

2. Clinical Perspective – Vivek Hande

2.1 Why Quality Control Matters

  • Described AI without QA as “modern‑day sorcery” – powerful but potentially hazardous.
  • Cited that ∼600‑700 AI‑enabled medical devices have FDA clearance, yet the majority remain un‑authorized in India, creating pressure to adopt unregulated tools.

2.2 Typical Quality Problems

Problem CategoryExample / Concern
Data IssuesBiased, incomplete, or non‑representative datasets
Model FitPoor suitability for the clinical workflow
UsabilityIntegration challenges, workflow friction
TransparencyBlack‑box decision making
Cyber‑security & PrivacyPotential data breaches, national‑security implications

2.3 Core Dimensions of AI Quality

  1. Clinical Effectiveness & Safety – must improve patient outcomes without causing harm.
  2. Robustness & Fairness – resistance to distribution drift; mitigation of bias.
  3. Privacy & Security – compliance with data‑protection regulations.
  4. Transparency & Accountability – explainability, audit trails, clear responsibility.

2.4 Lifecycle‑wide Quality Framework

  • Design & Development – rigorous data governance, bias audits.
  • Validation – clinical trials, local validation, reproducibility checks.
  • Deployment – human‑in‑the‑loop oversight, role‑separation (AI assists, clinicians decide).
  • Post‑deployment Monitoring – continuous performance tracking, automatic “kill‑switch” when drift detected.
  • Feedback Loops – systematic retraining with new, validated data.

2.5 Practical Implementation Blueprint

  • AI Governance Policy – slated for release by the Health Minister (within the hour).
  • AI Inventory – register all AI tools (embedded software, devices, cloud services).
  • Standardized Procurement – start with small, locally‑validated pilots before scaling.
  • Capacity Building – develop expertise in data governance, cybersecurity, and clinical validation.

Key Takeaways

“No AI without QA is the bottom line – poor‑quality AI can harm patients, exacerbate inequities, and expose organisations to severe regulatory and reputational risk.”

3. Regulatory & Operational View – Taruna Madan Gupta (ICMR)

3.1 “Kill‑Switch” Definition

  • Regulatory stance: An AI tool is considered a product only after it stops learning; the licensed version is immutable. Continuous learning occurs only in a parallel data‑collection pipeline that generates a new version which must undergo fresh regulatory approval. Hence the “kill‑switch” is built‑in by law – the deployed product never changes after release.

3.2 Multi‑Layered Quality Controls

  • Cyber‑security: Mandatory safeguards to protect patient data and national security.
  • Regulatory Framework: CDSCO draft (October) open for stakeholder feedback; final framework expected in a few months.
  • Continuous Performance Monitoring: Integrated into AI tools via the NIRMAN framework (developed by a six‑IIT Centre of Excellence).

3.3 Real‑World Validation Network

  • 84 ICMR‑affiliated clinical‑trial sites across disease‑specific endemic zones (e.g., malaria).
  • 170 medical colleges with multidisciplinary research units and rural health research units for field feasibility.
  • Health‑Technology Assessment (HTA): Post‑licensing comparative RCTs against standard of care; results feed into reimbursement and scaling decisions.

3.4 Outcome‑Based Evaluation

  • Emphasised moving beyond conventional performance metrics (accuracy, F1) toward patient‑outcome impact and cost‑effectiveness (ROI).

4. Accountability & Open‑Source Debate – Nisheeth Srivastava (IIT Kanpur)

4.1 Accountability Chain

  • Institution‑level responsibility: The health‑care institution (clinic, hospital, government facility) that signs off on an AI‑assisted prescription bears the ultimate liability, regardless of the vendor or model.

4.2 Incentives vs. Punishments

  • Balanced approach required:
    • Incentives (grant funding, fast‑track approvals) to encourage institutions to adopt robust QA practices.
    • Punitive measures (fines, licence withdrawals) for repeated safety breaches – especially important for hospitals, less so for individual clinicians who are scarce resources.

4.3 Open‑Source vs. Version‑Controlled Releases

  • Open‑source benefits: Crowdsourced control samples, collective improvement, especially for rare diseases where a single hospital cannot gather enough positives.
  • Risks: Lack of accountability, potential data leakage, uneven validation.
  • Suggested a hybrid model: open‑source data repositories (e.g., curated control sets) combined with regulated, version‑controlled production releases.

4.4 The “Forensics” Question (later addressed by Prof Gandhi)

  • Proposed that AI decision‑making should be audit‑friendly (traceability of model version, data inputs) rather than relying solely on post‑hoc forensic analysis.

5. Role‑Separation & Clinical Support – Tapan K Gandhi (IIT Delhi)

5.1 Use‑Case: Expert‑Level Training

  • AI can replicate expert gaze patterns (eye‑tracking) to teach novices (e.g., radiology) when senior clinicians are scarce.

5.2 Use‑Case: Contouring in Radiation Oncology

  • AI provides real‑time nudges during peer‑review of tumor contouring, restoring engagement lost in virtual review environments.

5.3 Role‑Separation Framework

  • AI‑assist zones: High‑volume, low‑complexity tasks (screening, triage) where AI reduces clinician workload.
  • Human‑expert zones: Complex, rare, or high‑stakes decisions where clinicians retain final authority.

5.4 Emphasis on Incremental Deployment

  • Stressed that AI is a general‑purpose technology that must be deployed cautiously, with clear scientific development pipelines and continuous validation.

6. Audience Interventions & Clinician Concerns

ParticipantConcern / CommentKey Point
Senior clinician (unnamed)Trust vs. “artificial” perception; fear of replacing human touchAI should assist not replace; patient‑physician relationship remains central
“Devil’s advocate” clinicianModel drift vs. disease drift; risk of medical education erosionNeed local validation and to avoid over‑reliance that erodes clinical skill
Educationist (unnamed)Cognitive debt from “spoon‑feeding” AI knowledgeCaution against over‑automation of learning; maintain deep reasoning
AI entrepreneur (unnamed)Difficulty obtaining clinician‑provided labelled data; incentives for doctorsICMR’s MedTech Mitra portal offers financial/technical support; encourages data sharing under regulated framework
Panelist (bonus question)Forensic vs. regulatory pathway for AI decisionsConsensus: AI must be transparent by design, with regulatory oversight; forensic analysis alone is insufficient

7. Announcements & Forward‑Looking Statements

  • AI Governance Model for the nation to be released by the Health Minister within the hour of the session.
  • CDSCO draft regulatory framework (opened October) seeking stakeholder comments.
  • NIRMAN – continuous monitoring framework under development by a six‑IIT Centre of Excellence.
  • MedTech Mitra – a “MedTech highway” portal (launched Dec 2023) offering innovators end‑to‑end support (financial, technical, regulatory).
  • Future Roadmap: Phase‑wise rollout – proof of concept → prototype → product; with mandatory post‑deployment audits and local validation at each stage.

See Also:

Graph View

Backlinks