Johnattan's Laboratory
← All experiments

Practising medical Spanish by talking out loud with a simulated patient

Clinicians with Spanish-speaking patients need to practise the conversation itself, out loud. I built Consulta, a voice app where you talk with a fictional patient and then get feedback on your Spanish.

  • Voice
  • Health
  • Education
  • Tools

Project page

Updated26 Sep 2026

The problem

The first user is a physician whose patients are largely Latin American and Caribbean, and who wanted to improve his Spanish. The practice had to work like a visit: you speak, the patient answers out loud in her own variety of Spanish, and afterwards someone tells you how your Spanish did. The patient should behave like a real one, such as a 62-year-old Dominican woman who says "me da una tembladera" and keeps her real worries to herself until she trusts you.

What was out there

Finding the voices was the hardest part. The free Microsoft Edge neural voices sounded robotic and regionally neutral. The voice data I could legally clone from, Mozilla Common Voice's Spanish spontaneous-speech set, has 332 clips from 12 speakers and none from the Caribbean, the Dominican Republic or El Salvador. Gemini's voice library has only Mexican and Argentine Latin American voices. ElevenLabs has about a dozen Dominican Spanish voices and no Salvadoran one. Gemini can replicate a real person's voice, but it needs 10 to 30 seconds of that person and the same person reading a consent statement, so building one from a synthetic sample would fake the consent check. The honest route is recording real Dominican and Salvadoran speakers with a written agreement, which I have not done yet.

What I built

Consulta, live at consulta.decilion.com, has two fictional patients: María, 62, a Dominican woman at a diabetes follow-up, and Rosa, 54, a Salvadoran woman with chest pressure when she walks fast. You read the chart, tap the microphone, speak Spanish, and the patient answers out loud. When you end the visit, a coach reviews the transcript.

The patient is the language test. Each one has hidden facts: María, for example, skips her morning pill because her sugar drops, drinks sodas to get through 12-hour caregiving shifts, and fears insulin because her mother died after starting it. The patient is told to share these only when the Spanish is warm, open-ended and respectful, and at most one per turn. Questions that sound translated from English, rushed or judgemental get a short, vague answer. She describes her body in everyday words ("una apretazón", "se me corre al brazo") and never in clinical terms, so the doctor has to do the decoding. She stays on topic and resists attempts to make her break character. You choose whether she calls you "doctor", "doctora" or neither.

The coach gives feedback in five areas (comprehensibility, vocabulary, register and culture, disclosure, and limits and repairs) plus an overall practice score. Every critique has to quote what the doctor actually said. Missing clinical topics never lower the score, and the app says plainly that the score has not been validated against human raters.

Patient speech is the part I kept changing:

Date Patient voice What happened
2026-04-22 Mistral Voxtral, cloned from a CC0 Common Voice clip Replaced Edge TTS. Natural, but the speaker is not Dominican
2026-05-22 OpenAI gpt-4o-mini-tts tested I found it "too generic"; Voxtral stayed
2026-06-14 ElevenLabs Flash v2.5, one voice per patient First per-patient voices
2026-09-24 ElevenLabs v3 Conversational Current voice on the public site (version 1.3.1)
2026-09-25 Gemini 3.8 Flash-Lite TTS, designed voice per patient Built and running on a preview branch only
Consulta's scenario picker in dark mode, headed 'Who are you seeing today?', with two patient cards: María Altagracia, 62, from the Dominican Republic, at a diabetes follow-up, and Rosa Elvira, 54, from El Salvador, with chest pressure when she walks fast.
The scenario picker: each fictional patient comes with her country, the reason for the visit and her chief complaint in her own Spanish.
The chart for the fictional patient María Altagracia Fernández: chief complaint in Spanish, nurse intake note, problem list, vitals, allergies and active medications, with a 'Begin consultation' button.
Before the visit you read a chart, as you would in clinic; the patient keeps the rest of her story to herself until you ask well.
The practice screen before the encounter starts: the patient card, goals for your Spanish, vitals and flagged labs on the left, and an empty conversation area with the prompt 'Greet María in Spanish' and a 'Tap to talk' button.
The visit screen waits for you: goals for your Spanish sit beside the conversation, and nothing is recorded until you tap the microphone.

Equipment

How I built it

  1. Before writing any scenario, I built a bare test rig (speech to text, model reply, text to speech, playback) with a timer on each step, because latency was the biggest risk. It showed that gpt-5.4-mini spent time reasoning before every reply, so the live patient moved to gpt-5.4-nano at low reasoning effort.
  2. I wrote each patient as data: chart, dialect rules, backstory, hidden facts and behaviour notes. The patient prompt and the coach rubric are built from the same file, so what the patient rewards is what the coach looks for. Hidden facts and the checklist stay on the server and never reach the browser.
  3. I grounded the coach in medical-Spanish education research, then removed claims the studies did not support (an old mapping of the score to a language-proficiency scale, and claims that what the patient discloses proves the effect of the doctor's Spanish). Feedback is capped per section so it stays short.
  4. For voices, I ran a blind A/B test: seven lines, each spoken by ElevenLabs v3 and by Gemini, with the order hidden. Then I benchmarked Gemini Flash and Flash-Lite on the paid tier: 15 lines per patient on each model, 60 clips in all, with a speech-to-text check of every clip against its line.
  5. When I heard the accent change from turn to turn, I checked the server logs before touching anything, then scored each Gemini clip against the patient's average voice with Resemblyzer speaker embeddings.
  6. The recent releases each had an independent review by a second model. Claude Fable 5.1 reviewed the reliability release and the move to ElevenLabs v3. Codex reviewed the Gemini provider and found four hardening gaps (an unbounded response, an incomplete audio check, which audio block to read, untested timeouts). I fixed all four.

Results

  • In the blind A/B test I preferred Gemini 3.8 Flash TTS over the ElevenLabs v3 voices on 7 of 7 lines. I then chose Flash-Lite over Flash while the app is in test mode, because it is cheaper and to my ear only slightly below Flash.
  • On Gemini's paid tier (30 lines per model), Flash-Lite took a median of 3.9 s with a slowest line of 5.1 s, and Flash a median of 4.7 s with a slowest of 6.8 s. There were no failures and no repeated lines. On the free tier the same calls took 4 to 180 s and one line was spoken twice.
  • A full turn on the preview, with real calls, took 1.2 s for speech to text, 2.8 s for the patient reply and 2.9 to 3.7 s for the Gemini voice.
  • List prices per 1,000 characters of patient speech are US$0.050 for ElevenLabs v3 Conversational, about US$0.020 for Gemini Flash and about US$0.014 for Flash-Lite. Google's prices double on 2027-01-01.
  • Every speech request in the window when I heard the accent change went to the public site and ElevenLabs v3; the Gemini preview received none. So the drift was most likely ElevenLabs v3, not Gemini.
  • Against the patient's average voice, every full Gemini sentence scored 0.90 or higher, and Flash was no steadier than Flash-Lite. Only one-word replies ("Sí.") scored lower. Embeddings measure timbre, not accent, so the accent still needs a human ear.
  • The public version has 93 automated tests and the Gemini branch has 104. Both pass with providers mocked, so tests spend no credits.

What I learned

Look for the bug where the user experiences it. I spent two rounds blaming the audio files for the stutter because a silence detector flagged long pauses. They were natural speech pauses; the files played perfectly on their own. The real cause was the browser re-rendering the page five times a second during playback.

Check the logs before trusting your ears, or a model. The accent drift I heard was on the live voice, not on the new one I was testing, and the logs settled it.

A free tier is not a benchmark. Gemini on the free tier took up to 180 s per line and once spoke a line twice; on the paid tier Flash-Lite's slowest line took 5.1 s.

Lab notes

  1. I built Gemini 3.8 Flash-Lite TTS with a designed voice per patient on a preview branch, benchmarked 30 lines on two models and moved the app to consulta.decilion.com. Server logs then showed that an accent change I heard came from the live ElevenLabs v3 voice, and speaker embeddings checked the Gemini voices.

  2. I hardened the app for version 1.3.0 and moved patient speech to ElevenLabs v3 Conversational, then ran a blind A/B test in which I picked Gemini 3.8 Flash TTS on 7 of 7 lines.

  3. I traced the audio stutter to the browser redrawing the transcript every 220 ms during playback and removed that redraw. Then each patient got her own ElevenLabs Flash v2.5 voice, at about 390 ms through my route.

  4. With Claude Code I built a bare latency test (voice to transcript to patient reply to speech, timed at each step), then the first patient, her chart and the coach, and put it online. The next day I rebuilt the coach to score only the Spanish, leaving clinical coverage out, and added a second patient.

© 2026 Johnattan García-Ruiz Last updated