India AI Impact Summit 2026

governing-scientific-ai-at-scale-safety-evaluation-and-institutional-readiness

7 min read

TRACK:Science
SPEAKERS
  • Dr. Geetha Raju, CeRAI - IIT Madras
  • Dr. Shyam Krishna, RAND Europe
  • Dr. Suryesh Kumar Namdeo, Indian Institute of Science, Bangalore
  • PT Nhean, AI Safety Asia
SESSION OVERVIEW

As capabilities scale, existing assurance and risk management approaches are often stretched by real-world scientific and institutional contexts. This session examines how safety, evaluation, governance frameworks can be adapted and stress-tested through scientific AI use cases, and how emerging ecosystems can build evaluation capacity, manage high-impact risks, and contribute to globally robust scientific AI safety practices.

VIDEO RECORDING

Governing Scientific AI at Scale: Safety, Evaluation, and Institutional Readiness

Abstract

The panel examined how the rapid scaling of AI‑enabled scientific tools strains traditional risk‑management frameworks that were built around physical laboratory controls. Speakers argued that safety, evaluation, and governance must move upstream—into the design phase of AI‑driven bio‑/chem‑/nuclear research—while also being adapted to the heterogeneous capacities of institutions across the Global South. Key themes included the proliferation of AI‑bio‑design tools, the need for tiered access models, decentralized “web‑of‑prevention” safeguards, systematic red‑team / independent evaluation, and harmonised data‑standard and legal “safe‑harbor” mechanisms for cross‑border biosurveillance.

Detailed Summary

Dr. Geetha Raju opened the session (noting she is not an AI‑safety specialist but a biosecurity researcher). She described a structural transformation in life‑science risk governance:

  • Historically, risk control was tied to physical infrastructure – lab inspections, material‑transfer agreements, containment protocols.
  • AI‑driven bio‑design tools (protein‑engineering, DNA‑optimization, pathogen‑host modelling) have decoupled risk from the lab.

She cited a RAND study estimating >1,500 AI‑bio‑design tools now in circulation, radically altering how scientific work is performed. Because AI models can be accessed remotely, the risk landscape has moved upstream, to the design stage, before any physical containment can be applied.

Key points:

  • Data governance, model evaluation, and red‑team exercises remain vital, but must be augmented by new institutional mechanisms.
  • India’s vibrant yet uneven scientific ecosystem demands capacity‑building for AI‑bio‑security, AI‑chemical, and AI‑nuclear domains.

2. Institutional Readiness & Decentralised Oversight

2.1 Leveraging Existing Institutional Units

Geetha highlighted that many Indian research institutes already possess Information‑Security Offices or Biosafety/Biosecurity Officers. She argued these units could be up‑skilled to handle AI‑related threats:

  • Integrate AI‑evaluation modules into existing biosafety audit workflows.
  • Transform periodic, paper‑based inspections into continuous, AI‑augmented monitoring.

2.2 Need for Distributed Governance

She warned against a single, centralized authority in Delhi: such a model would be overwhelmed by the speed of AI development. Instead, a decentralised “web‑of‑prevention”—multiple, overlapping safeguards—should be cultivated, echoing classic bio‑security strategy where no single control is sufficient.

3. Open Science, Tiered Access, and Governance Norms

PT Nhean steered the conversation toward the tension between open‑science benefits and the danger of “destabilising diffusion” of high‑risk capabilities:

  • Binary decisions (open vs. closed) are unrealistic.
  • Proposes a tiered‑access model:
    • Credentialed researchers (e.g., defensive labs, medical‑countermeasure groups) receive privileged access.
    • Open‑source tools remain available, but with responsible‑use guidelines and monitoring.

She likened the approach to “Know‑Your‑Customer” (KYC) practices in finance, arguing that pre‑deployment assessments—structured rubrics evaluating downstream risk—are essential.

4. RAND Europe’s Risk‑Index & Pre‑deployment Rubrics

Dr. Shyam Krishna presented RAND Europe’s work:

  • Development of a Global Risk Index for AI‑enabled biological tools—cataloguing tools, their capabilities, and associated threat vectors.
  • Implementation of a pre‑deployment assessment framework:
    • Uses structured rubrics (capability, dual‑use potential, mitigation measures).
    • Mirrors KYC: credential verification for high‑risk research teams before tool release.

He emphasized that once a frontier model is released, the danger is already “out there”; therefore, preventive assessment is the only viable safety lever.

5. Technical‑Policy Gaps in the Indian & South‑East Asian Context

Dr. Suryesh Kumar Namdeo outlined several institutional and technical gaps:

5.1 AI‑Readiness Disparities

  • India ranks third globally in AI research output, but many South‑East Asian nations lag (e.g., Indonesia ~49th).
  • Existing AI policies often import western‑centric benchmarks that ignore local socio‑cultural specifics.

5.2 Benchmark Failures

  • A South‑East Asia safety benchmark revealed that 20‑30 % of leading LLMs failed to mitigate biological‑risk scenarios.
  • Calls for socio‑cultural evaluation that reflects local deployment contexts (language, health infrastructure, socioeconomic factors).

5.3 Participatory, Bottom‑Up Governance

  • Advocacy for participatory approaches that involve end‑users, local stakeholders, and domain experts in the requirements‑definition phase.
  • Suggestion to prioritize small‑language models for edge deployments in low‑resource settings, rather than defaulting to large, resource‑intensive LLMs.

5.4 Policy Alignment

  • India’s government promotes self‑regulation and voluntary commitments for AI risk management.
  • Recommends a unified, adaptable framework that can be calibrated for diverse deployment environments.

6. Institutionalising Independent Evaluation & Red‑Team / Monitoring

6.1 Analogies to Nuclear Governance

Geetha drew a parallel with the International Atomic Energy Agency (IAEA): nuclear materials are scarce, traceable, and heavily regulated, whereas biological data and AI tools are diffuse, dual‑use, and commercially accessible.

6.2 Evidence of Model Superiority

Citing a Secure Bio study, she noted ChatGPT‑3 outperformed expert virologists by 94 % on troubleshooting virology protocols—a startling illustration of the speed at which AI can surpass human expertise.

6.3 Six‑Monthly Global Monitoring Ritual

  • RAND Europe recommends semi‑annual, coordinated risk‑monitoring by governments and independent researchers.
  • AI‑automation can scale the monitoring process, but a multilateral investment is required to sustain it.

6.4 Proposal for an AI Safety/Security Institute

  • An independent, credentialed institute embedded with a formal governmental liaison (potentially anchored to the Biological Weapons Convention or WHO) would:
    • Conduct pre‑deployment assessments.
    • Share results through a tiered‑confidentiality network (not kept proprietary).

7. Building Capacity in the Global South

Geetha and PT Nhean emphasized the need to train a larger cohort of professionals in AI‑bio‑security, AI‑chemical‑security, and AI‑nuclear‑security.

  • AI Safety Asia (PT Nhean) is establishing regional training programs, incubating AI safety institutes akin to the AGITA home‑institute at IIT Madras.

  • Collaborative networks (e.g., a Global‑South Trustworthy AI network) will facilitate knowledge sharing and benchmark development tailored to low‑resource settings.

8. Harmonising Data Standards & Cross‑Border Surveillance

PT Nhean transitioned to a discussion on AI‑enabled biosurveillance:

  • Current systems suffer from incompatible data standards and fragmented legal regimes, hindering rapid, coordinated response (exemplified by COVID‑19 data hoarding).

  • Proposed solutions:

    • Federated standards (e.g., adapting HL7 FHIR for public‑health surveillance).
    • Pre‑negotiated legal “safe‑harbors” for cross‑border data sharing during emergencies.
    • Shared evaluation criteria that respect national contexts yet enable interoperability.

She warned that AI governance and bio‑security communities often operate in silos, creating gaps where risk materialises.

9. Closing Reflections & Future Directions

  • Suryesh stressed that model‑centric metrics (performance, bias) are insufficient; we must assess socio‑technical readiness—funding structures, publication incentives, startup ecosystems, and incident‑response capacity.

  • Geetha highlighted the digital‑to‑physical barrier: even if AI tools are controlled, physical synthesis of pathogens still requires hardware safeguards.

  • Shyam reiterated the value of agentic AI for detecting misuse (e.g., jailbreak attempts) and the importance of continuous learning loops where risk incidents feed back into model training.

  • PT Nhean concluded with a call for decentralised yet integrated leadership, empowering biosafety officers and institutional committees, and establishing clear national–regional coordination (citing Singapore’s multi‑agency model) as a template.

Key Takeaways

  • Risk has moved upstream: AI‑driven bio‑design tools shift the safety frontier from physical labs to the design phase, demanding new governance mechanisms.
  • Decentralised “web of prevention”: No single authority can oversee all AI‑enabled scientific work; overlapping safeguards across institutions are essential.
  • Tiered access & pre‑deployment rubrics (RAND’s KYC‑style framework) provide a pragmatic middle ground between open science and blanket restriction.
  • South‑Asian AI readiness is heterogeneous; policies must incorporate socio‑cultural benchmarks and support small‑model, edge‑deployment strategies for low‑resource settings.
  • Independent, credentialed AI safety institutes (potentially anchored to IAEA/WHO) can coordinate semi‑annual global risk monitoring and disseminate assessment results through tiered‑confidential channels.
  • Capacity‑building in AI‑bio‑security, AI‑chemical‑security, and AI‑nuclear‑security is urgently needed across the Global South, with regional hubs such as AI Safety Asia leading training efforts.
  • Data‑standard harmonisation and pre‑negotiated legal safe‑harbors are critical for effective, cross‑border biosurveillance and to avoid the fragmentation that hampered COVID‑19 responses.
  • Model‑centric metrics are insufficient; evaluation must also capture institutional readiness, funding incentives, and incident‑response mechanisms.
  • Agentic AI can serve as an early‑warning system, detecting misuse attempts and feeding incident data back into model improvement pipelines.
  • Collaboration between AI‑governance, bio‑security, and public‑health communities must be institutionalised to close the “silo” gap that currently leaves the system vulnerable.

See Also:

Graph View