Safe AI Solutions in Education - A Practitioner-oriented Dialogue for the Global South Perspective

Abstract

The panel examined what “safe” AI means when it is embedded in K‑12 and higher‑education ecosystems, especially in low‑resource contexts across the Global South. Drawing on a six‑layer “co‑intelligence” architecture, the discussion covered design safeguards in Khan Academy’s AI tutor (Kanmigo), risk perceptions among the audience, challenges of AI‑augmented assessment and admissions, methods for auditing fairness, shared accountability across data curators, model developers and deployers, trade‑offs between accuracy and reach, and how large‑scale AI literacy programmes can empower teachers, students and parents. The session closed with practical criteria for funders deciding which AI‑in‑education interventions merit scaling.

Detailed Summary

Krishnan Narayanan opened the session, thanking the audience and introducing the Center for Responsible AI (CIRA) at IIT Madras. He described CIRA as a multidisciplinary non‑profit that bridges technical research and policy work to produce accountable, explainable, and trustworthy AI for the Global Majority.

He then introduced the panelists, likening the mix of viewpoints to a “thali” (an Indian platter) and noting that each participant would keep answers short and sweet to allow ample audience interaction.

2. The “Co‑Intelligence” Lens

Anil Ananthaswamy presented his four‑year research on co‑intelligence, a socio‑technical architecture consisting of six layers:

Layer	Focus
1–3 (Infrastructure)	Data, model training, agentic systems
4 (Interaction)	Human–AI dialogue
5 (Life‑experience)	Learner, teacher, administrator contexts
6 (Ecosystem)	System‑wide learning & insight sharing

He emphasized that safety must be evaluated across all six layers, not just at the algorithmic level. This framing guided the rest of the discussion.

3. Safe‑by‑Design in Khan Academy’s “Kanmigo”

Swati Vasudevan (Khan Academy India) responded to the moderator’s opening question, “What does safe‑by‑design look like for Kanmigo?”

Key design safeguards she highlighted:

1. Curated Content – Kanmigo draws only from Khan Academy’s verified, curriculum‑aligned library; no ad‑hoc internet scraping.
2. Privacy‑First Architecture – The platform does not collect personally identifiable information (PII); thus the AI tutor never accesses student IDs or names.
3. Interaction Logging & Risk‑Flagging – Every student‑AI exchange is logged. Automated analytics detect potentially risky or self‑harm‑related statements and alert the supervising teacher.
4. Human‑in‑the‑Loop Deployment – The rollout in India is teacher‑directed; teachers retain supervisory control and can intervene or disable the tool.

She underscored a step‑wise pilot approach: small pilots → learning from failures → capacity‑building → large‑scale rollout.

Audience follow‑up – A participant asked whether Indian safety constraints differ from the U.S. Swati replied that the risk profile of students is universal; the same guardrails work in Latin America, the U.S., and India. She cited a self‑harm detection case in a Latin‑American pilot as proof of cross‑regional applicability.

4. Audience Poll: Perceived Biggest Risk

The moderator displayed a live poll with several risk statements (student dependency, reduced critical thinking, data leakage, etc.). ≈70 % of respondents selected “Student dependency / reduced thinking” as the top concern.

The result set the stage for the next speaker’s deep‑dive on higher‑education risks.

5. Higher‑Education Viewpoint – AI Fluency & Safety

Srinivasan Parthasarathy (Ohio State University) explained Ohio State’s “AI Fluency” initiative, which aims to expose all 60 000 students across 100+ disciplines to at least one AI‑related module before graduation.

He identified three safety‑related challenge domains:

1. Admissions Manipulation – Students, recommenders, and admission officers are already employing LLMs to draft personal statements, recommendation letters, and AI‑driven candidate screening. This creates a feedback loop where AI evaluates AI‑generated artifacts, eroding authenticity.

2. Classroom Assessment Integrity – Open‑book exams now require multi‑model vetting of questions to avoid immediate LLM solving. The pace of model improvement means exam designers must constantly invent new “clues” to stay ahead.

3. Cognitive & Mental‑Health Impact – Excessive reliance on AI for peer‑like query answering (e.g., “ask Gemini”) may diminish collaborative learning, critical reasoning, and mental‑well‑being—parallel to concerns raised about social‑media overuse.

He also referenced a recent Nature paper suggesting that LLM‑assisted grant writing tends to produce low‑risk, “safe‑bet” research proposals, potentially stifling high‑risk innovation.

Key point: Safety in higher education means guardrails that preserve human judgment, keep assessments robust, and safeguard cognitive development.

6. Auditing AI Systems – The AWAR Perspective

Srinivasan Parthasarathy (continuing) introduced AWAR, a tool he co‑developed for online fairness auditing. He argued that auditing must be context‑aware: the same model may be safe for a kindergarten classroom but unsafe for an advanced research lab.

Audit challenges discussed:

• Defining “safe” a priori is difficult; safety often emerges only after deployment.
• Regulatory fairness metrics (e.g., demographic parity) are easier to formalize than holistic safety thresholds.
• Human‑in‑the‑loop: auditing should also evaluate teacher and student behaviours, not just the algorithm.

He concluded that continuous, multi‑layer monitoring—aligned with the co‑intelligence layers—is essential.

7. Accountability & Liability – Distributed Responsibility

A rapid round asked each panelist to name who should be held accountable if an AI tutor harms a school. Responses converged on shared responsibility, with nuance on liability:

Panelist	Core Message
Anil Ananthaswamy	Responsibility spans data curators, model developers, and deployers; “all of the above” but liability is case‑by‑case.
Shaveta Sharma‑Kukreja	Strategic designers & deployers carry ultimate liability because they approve scale‑up; concrete accountability lies with the entity authorising deployment (state education department or private school leadership).
Swati Vasudevan	While everyone is responsible, deployment architects must embed checks‑and‑balances; teachers remain a crucial safety layer but cannot shoulder full liability.
Sunil Wadhwani (philanthropic perspective)	Emphasised risk‑benefit calculus: safety is essential but cannot delay life‑changing interventions. Liability rests with the organisation that signs off on a system’s rollout.

The discussion highlighted the tension between collective duty and pinpointed legal liability, especially in publicly funded school systems.

8. Trade‑Off: Accuracy vs. Reach

An audience poll asked: “Would you accept an AI tutor that is 5 % wrong but reaches 10× more children?”

• Panel consensus: Yes, provided robust human oversight.
• Swati stressed that teacher supervision can catch the occasional error, preserving the net benefit of massive reach.
• Anil warned that a 5 % hallucination rate could erode trust if not transparently reported.

The exchange reinforced the “human‑in‑the‑loop” motif as the mitigating factor for imperfect models.

9. Global‑South Lens – Pedagogy, Access, and Trust

Shaveta Sharma‑Kukreja (CSF) articulated a four‑point safety framework for the Global South:

1. Pedagogical Alignment – AI tools must respect curriculum, grade‑appropriateness, and sound instructional practices.
2. Age‑Appropriate & Content‑Appropriate – Tools should not expose young learners to unvetted content.
3. Transparency & Assistive Role for Teachers – AI should augment teachers, never replace them.
4. Systemic Integration – Safety must be baked into school governance, teacher capacity‑building, and accountability mechanisms.

A brief debate emerged over whether access constraints (e.g., limited device time, low digital literacy among parents) demand different safety norms. Swati counter‑argued that access and safety are orthogonal: scaling access does not diminish the need for strict safety standards.

Shaveta added that language localization (vernacular interfaces) is an access‑driven safety measure: if students can interact in their mother tongue, the chance of misunderstanding and misuse drops.

Anil highlighted that illiterate parents still need a simple visual cue (e.g., a logo) to recognize safe AI interactions at home.

10. AI Literacy Initiative – “AI Samarth”

Shaveta described CSF’s AI Samarth programme, a nation‑wide AI‑literacy effort in partnership with IIT Madras.

• Scope: 1 million students and teachers across 10 Indian states have received introductory modules on what AI is, how it is already embedded in their learning journeys, and how to become responsible users.
• Rationale: With 82 % of 14‑16‑year‑olds using smartphones for learning, many are already passively interacting with AI (e.g., search‑engine suggestions). Turning them into active, informed agents reduces the likelihood of harm.
• Pedagogical Core: Emphasises the Socratic method, critical questioning and awareness of model limitations (no error‑rate reporting, lack of human‑in‑the‑loop visibility).

She argued that AI literacy is the prerequisite for any safety‑by‑design roll‑out, because a trained human can better supervise, question, and intervene.

11. Data‑Logging Debate

The moderator asked whether schools should retain logs of student–AI conversations for safety audits.

• Srinivasan highlighted the utility of logs for personalised learning analytics but warned about privacy and access control.
• Swati advocated for anonymised, minimised logs that teachers can review when needed, but insisted on strict governance around who can view them.
• Anil raised the potential chilling effect on student expression if logs are perceived as surveillance.

The panel concluded that log retention is advisable but must be privacy‑first, limited in scope, and governed by clear policies.

12. Funding Decisions – Pragmatic Criteria

Sunil Wadhwani (founder, Wadhwani AI) shared the philanthropic lens for evaluating AI‑in‑education proposals:

1. Government Priority Alignment – Is the problem in the top 3–4 priorities of central or state governments?
2. AI‑Suitability – Does the problem have high‑quality data and a clear AI solution path?
3. Scalability Blueprint – Can the solution reach the last mile (teachers, frontline workers, parents) and integrate with existing systems?

He illustrated these criteria with a case study from Gujarat: an AI‑driven oral‑reading‑fluency platform that now serves 3 million children across Gujarat and Rajasthan. Although the system may not pass every fairness audit perfectly, the benefits (dramatic dropout reduction) outweigh the residual risks.

Sunil warned against over‑engineering for perfection: “waiting five years for a perfect model means millions stay out of school today”.

13. Closing Reflections

The moderator summarized the key themes that surfaced:

• Human‑in‑the‑loop as the cornerstone of safety across all layers.
• The need for frameworks that address fairness, bias and ecosystem governance.
• Co‑agency: empowering digitally unsophisticated learners through teacher mediation.
• Scale‑inclusive infrastructure – recent announcement of India’s Center of Excellence for AI in Education (led by IIT Madras) to create standards, policies and interoperable data exchanges (DPIs).

A warm round of applause concluded the session.

Key Takeaways

• Safety must be evaluated across six co‑intelligence layers (data → models → agents → interaction → life‑experience → ecosystem).
• Khan Academy’s Kanmigo exemplifies safe‑by‑design: curated curriculum, privacy‑first data handling, automated risk‑flagging, and teacher‑directed deployment.
• Higher‑education risks include AI‑generated admissions material, compromised exam integrity, and potential cognitive decline from over‑reliance on AI assistants.
• Auditing tools (e.g., AWAR) need to be context‑aware; safety cannot be fully specified before real‑world use.
• Accountability is shared among data curators, model developers, and especially the entity that authorises large‑scale deployment; liability is case‑by‑case.
• A 5 % error rate is acceptable if the system reaches many more learners and is supervised by teachers.
• Global‑South safety hinges on pedagogical alignment, language localisation, transparent teacher augmentation, and system‑wide governance.
• AI literacy (AI Samarth) is essential: turning passive users into informed agents reduces misuse and improves safety.
• Retention of anonymised interaction logs aids auditing but must respect privacy and be governed by clear access policies.
• Funders should prioritize: (1) alignment with government priorities, (2) AI‑technical suitability, and (3) demonstrable scalability to the last mile.
• Human‑in‑the‑loop remains the cornerstone of trustworthy AI in education, ensuring that technology augments rather than replaces human judgment.

---

See Also:

• ai-2-0-reimagining-indian-education-system
• safe-and-trusted-agentic-ai-building-accountability-and-inclusion-for-india-and-the-global-south
• multi-stakeholder-collaboration-to-foster-ai-adoption-in-the-global-south
• empowering-the-human-edge-building-a-future-ready-workforce-in-the-age-of-ai
• enterprise-adoption-of-responsible-ai-challenges-frameworks-and-solutions
• ai-for-economic-growth-and-social-good-ai-for-all-driving-economic-advancement-and-societal-well-being
• ai-for-indias-next-billion-intergenerational-insights-for-inclusive-and-future-ready-growth