Deep Learning Indaba 2026 Workshop

Beyond Benchmarks: Scaling Multi-Turn Participatory AI Evaluations in Health and Education

Deep Learning Indaba 2026 Workshop

While traditional AI benchmarks are essential for baseline testing, they fall short in mapping exactly where models falter across precise use cases and diverse population segments. Co-organized by YUX Cultural AI Lab and Microsoft Research Africa, this hands-on workshop draws on health and education studies in Rwanda, Kenya, and Senegal to teach researchers how to bridge this gap using longitudinal diary studies.

Partners & Organizers

Co-created by research and deployment teams

YUX Cultural AI Lab

Kitala.ai

Microsoft Research Africa

About the Workshop

From static benchmark scores to grounded diagnostic evidence

Traditional AI benchmarks provide an essential baseline for model performance and laboratory validation, but they have stark limitations in real-world applications. They often fail in African health and education deployments because they measure static, single-turn responses rather than the iterative context-gathering required by actual users over time.

Co-organized by YUX Cultural AI Lab and Microsoft Research Africa, this interactive workshop uses concrete examples from deployments in Rwanda, Kenya, and Senegal to bridge this evaluation gap.

Our core objective is to provide researchers and practitioners with the practical infrastructure to map exactly where AI models fail across precise use cases and specific population segments.

Workshop Objectives

Practical evaluation methods for complex human settings

The Benchmark Breakdown

Analyze the strengths of automated baselines and the limitations of omitting dynamic human behavior, local linguistic nuances, and socio-cultural safety considerations.

Design Multi-Turn Scenarios

Collaborate using custom persona cards to create realistic multi-day evaluation dialogues tailored to different user groups.

Map Failures via Live Simulation

Use Kitala.ai to run evaluations, participate in simulations, and explore analytics dashboards.

Actionable Diagnostic Insights

Transform qualitative findings into structured evaluation insights and community-led standards.

Workshop Takeaways

Participants will leave with methods they can reuse immediately

Hands-on experience in large-scale human-in-the-loop evaluation
Practical methods for identifying hidden model alignment gaps
Experience designing multi-turn evaluation frameworks
Open-source templates and methodologies for future deployments
Knowledge applicable to health and education AI systems

Registration

Participant registration

Register your interest for the Beyond Benchmarks workshop and share the language, evaluation, and domain context that will help shape the session.

Full Name Email Address Affiliation/Organization Role/Background Primary Language(s) of Proficiency Other Languages Spoken

Experience with AI Evaluation

Beginner Intermediate Advanced

Areas of Interest

Healthcare Education Financial Services General/Other

Expectations for the Workshop

Workshop Agenda

A 90-minute hands-on program from critique to roadmap

Participants move from evaluation methods into scenario design, live testing, and shared analysis.

10 min
The Evaluation Gap
Led by Microsoft Research Africa.
- Review evaluation methods for AI systems, including the Agency Fund framework.
- Discuss the pros and cons of benchmarks in health and education contexts.
- Cover single-turn limits, contamination, lack of profile segmentation, and related risks.
- Identify challenges in human-based evaluation and when to use it, including Samiksha.
- Define stronger evaluation metrics across segments, methods, and use cases.
20 min
Co-Creation of Scenario & Metrics
- Introduce health and education scenarios and personas.
- Create groups of 3 to 4 by language or country.
- Brainstorm metrics such as contextual relevance, trustworthiness, and usefulness.
- Choose persona cards and build 3-turn dialogues from the scenario and cards.
- Add adversarial nudges such as ambiguous symptoms and code-switching.
- Share out scenario choices and metric priorities.
10 min
Technical Setup
- Walk through Kitala.ai prompts, data collection, and pairwise setup.
- Choose between pairwise testing, such as OpenAI GPT vs Google Gemini, or a single-model setup.
- Prepare text-based and image generation testing.
- Keep voice ready to showcase, while noting room-size constraints.
- Connect participant devices with one laptop per group.
30 min
Live LLM Testing
- Role-play personas in multi-turn scenarios.
- Rate GPT vs Gemini against each group's chosen metrics.
- Use facilitator prompts to test adversarial edge cases.
- Pause near the end for each group to note its top failure.
20 min
Collective Analysis & Roadmap
- Review the live dashboard and share failure modes.
- Identify patterns across personas, languages, and metrics.
- Discuss community-led evaluation standards.
- Define next steps to publish templates and results.