Methodology
How each metric on this page is computed, traced to the scoring code. All models are scored on the same held-out sets; the pipeline is a cascade of STT → LLM → TTS.
Datasets
- STT & TTS — Google FLEURS test split (English 647, Chinese 945, Malay 749, Tamil 591 clips).
- End-to-end pipeline — VoiceBench (English), a spoken-instruction benchmark with 7 subtasks.
- End-to-end pipeline — SEA-HELM (English, Malay, Tamil, Indonesian, Chinese — coverage varies per subtask), 4 subtasks adapted from the SEA-HELM text benchmark.
Text normalization (before WER/CER)
Both the reference and the hypothesis are normalized with the same pipeline before any error rate is computed — this removes cosmetic differences (case, punctuation, spacing, script variants) that would otherwise inflate WER/CER with false errors.
| Option | What it does | Effect |
|---|---|---|
normalize | Language-aware text normalizer — EnglishTextNormalizer for English; a script-aware Indic normalizer for Tamil (whisper-normalizer TamilNormalizer + punctuation strip — Whisper's BasicTextNormalizer would strip Tamil's combining vowel marks and shred words into characters); BasicTextNormalizer otherwise. Lowercases, strips punctuation, collapses whitespace. | ref + hyp |
remove_spaces | Drops all spaces (space-free scripts, e.g. Chinese). WER is undefined without word boundaries, so it is skipped — only CER is reported. | CER only |
zhconv | Converts Traditional → Simplified Chinese (zhconv zh-cn) so the two scripts don't count as errors. | zh only |
Per-language settings in use:
| Language | normalize | remove_spaces | zhconv | Reported |
|---|---|---|---|---|
| English | yes | no | — | WER + CER |
| Chinese | yes | yes | yes | CER only |
| Malay | yes | no | — | WER + CER |
| Tamil | yes* | no | — | WER + CER (both shown) |
*Tamil uses the Indic normalizer (canonicalizes the script, expands numbers, keeps combining vowel marks). Tamil WER is still inflated by compound-word segmentation — the same sounds written as one word vs. two (e.g. வேகவைக்கப்பட்ட vs. வேக வைக்கப்பட்ட) count as word errors — so the table shows CER alongside WER; externally published Tamil "WER" figures are usually character-level and comparable to our CER.
Edge case: if a string is empty after normalization, the sample scores 0.0 when both ref and hyp are empty, else 1.0 (rather than dividing by zero).
Source: scorers/common/transcription_scorer.py · configs/languages_config.yaml
STT — WER / CER
The STT model produces the transcript during generation; scoring runs no inference. WER and CER are computed with jiwer on the normalized reference vs. hypothesis:
WER = word_error_rate(ref_norm, hyp_norm) · CER = character_error_rate(ref_norm, hyp_norm)
TTFT (time-to-first-token) is reported per model: streaming models (e.g. ElevenLabs scribe_v2_realtime) stamp it at the first partial transcript; batch models report a run-level proxy since they can't stream.
The Overall column is the mean error rate across a model's languages — WER for space-delimited languages (including Tamil), CER for space-free ones (Chinese). Tamil additionally shows its CER in a separate column, since its WER overstates errors on compound segmentation.
Source: scorers/common/transcription_scorer.py · lib/evaluation-config.js (errValue)
TTS — intelligibility (WER/CER) + naturalness (UTMOSv2)
TTS is scored two ways:
- Intelligibility (WER/CER) — the generated audio is re-transcribed by the language's reference STT model (
whisper-large-v3), and that transcript is scored against the original TTS input text (same normalization as STT). Low WER/CER means the synthesized speech is clearly recognizable. - Naturalness (UTMOSv2) — see below.
- TTFA (time-to-first-audio) — how it's obtained depends on the model's serving mode:
- Streaming local (Qwen-TTS via vLLM) — measured per-clip as the wall-clock to the first audio chunk.
- Streaming cloud (ElevenLabs v3) — network-bound; reported as a per-language proxy measured on 10 streamed samples (~0.53–0.55s, Tamil ~0.80s).
- Non-streaming (OmniVoice, flow-matching diffusion) — the whole clip is denoised at once, so the first audible audio exists only when the full clip does: TTFA = total generation time (a proxy by construction, not a measured stream time; ~0.52–0.55s, nearly duration-independent).
Source: scorers/common/transcription_scorer.py (task_type="tts") · models/tts/elevenlabs/model/model.py
UTMOSv2 (naturalness MOS)
UTMOSv2 is a reference-free mean-opinion-score predictor: it estimates a naturalness score (~1–5, higher is better) from the audio alone — no reference text or audio needed. The pretrained model is built once and run over the whole directory of generated wavs, returning a predicted MOS per clip that is averaged per (model, language).
Source: scorers/tts/mos_scorer.py (utmosv2.create_model)
VoiceBench (end-to-end pipeline)
VoiceBench measures task success of the full cascade on spoken instructions. For each sample the LLM is scored on two arms:
- Text (topline) — the LLM answers the gold text prompt (no STT errors).
- Audio (cascade) — the LLM answers the STT transcript, so it inherits any STT mistakes.
The gap between them is the STT front-end's downstream cost on answer quality. Each subtask is scored differently:
| Subtask | Type | How it's scored |
|---|---|---|
| commoneval, alpacaeval | open | LLM judge (Claude Opus) rates the answer 1–5; averaged. Normalized ÷5 for the 0–1 aggregate. |
| sd_qa | qa | LLM judge gives Yes/No over n=3 samples; majority vote → correct (1/0). |
| mmsu, openbookqa | mcq | Rule-based: extract the chosen A/B/C/D from the response and match the reference → accuracy (1/0). |
| advbench | harm | Rule-based refusal detection: a safe refusal (or empty) scores 1.0 (higher = safer). |
| ifeval | ifeval | Strict instruction-following (VoiceBench's instruction_following_eval): 1.0 only if ALL instructions are satisfied. |
On a VoiceBench row of the Voice Agent leaderboard, Acc (text) and Acc (audio) are the macro-average of these per-subtask scores across all subtasks, for the gold arm and cascade arm respectively. Each score is normalized to 0–1 first — for VoiceBench that means dividing the gpt_1to5 judge ratings by 5; every other subtask metric here is already a 0–1 fraction.
Source: scorers/e2e/voicebench_scorer.py · scorers/e2e/external_libraries/voicebench/
SEA-HELM (end-to-end pipeline)
SEA-HELM adapts four tasks from the (text-only) SEA-HELM benchmark suite into a voice cascade: each prompt is synthesized to speech, transcribed back, and answered by the LLM — same Text vs. Audio two-arm design as VoiceBench above. Every subtask is scored rule-based (no LLM judge):
| Subtask | Measures | How it's scored | Languages |
|---|---|---|---|
| seahelm_ifeval | Instruction-following | SEA-HELM's own multilingual checker (word/sentence/paragraph counts, keywords, JSON, quoting, response language, ...) — pass/fail (1/0) per instruction, language-normalized. | en, id, vi, th, tl, ta, ms, my, km, lo |
| seahelm_global_mmlu_lite | Knowledge (multiple choice) | VoiceBench-style: extract the chosen A/B/C/D from the free-text response and match the reference → accuracy (1/0). | en, id, my, zh, vi, ms, tl |
| seahelm_question_answering | Extractive QA | SEA-HELM's own token-overlap F1 against the reference answer(s) (Thai/Chinese word-segmented first, since neither uses whitespace word boundaries). | id, vi, th, ta, zh |
| seahelm_safeguard | Safety (refusal rate) | Rule-based refusal-string detection on harmful instructions repurposed from SEA-HELM's own safety data (VoiceBench advbench's approach) — a safe refusal (or empty response) scores 1.0. | en, id, ms, my, ta, th, tl, vi |
Language coverage differs per subtask (a real gap in each task's own upstream data, not a bug) — a language missing a subtask's data simply contributes no samples to that one subtask, while the rest still score normally.
On a SeaHelm row of the Voice Agent leaderboard, Acc (text) and Acc (audio) are the macro-average of whichever subtasks have data for that pipeline/language, for the gold arm and cascade arm respectively. Each score is normalized to 0–1 first — ifeval's language-normalized accuracy is SEA-HELM's own 0–100 convention and is divided by 100, while the other subtask metrics are already 0–1 fractions.
Because the leaderboard rows on (pipeline, language, benchmark), a language covered by both benchmarks gets one VoiceBench row and one SeaHelm row for each pipeline, and a SeaHelm-only language gets a SeaHelm row and no blank VoiceBench one. The two accuracy columns therefore mean whatever that row's Benchmark cell says.
Source: scorers/e2e/seahelm_scorer.py · scorers/e2e/external_libraries/seahelm/
Pipeline latency (time-to-first-audio)
The pipeline latency is the perceived time from the user finishing speaking to the first audio of the reply — the sum of each stage's first-token / first-audio latency:
latency = STT TTFT + LLM TTFT + TTS TTFA
- STT TTFT — time to the first transcript token (streaming realtime STT).
- LLM TTFT — time to the first response token on the cascade (audio) arm, averaged across the subtasks of that row's benchmark.
- TTS TTFA — time to the first synthesized audio chunk (streaming).
Note: full generation times (whole transcript / whole reply / whole clip) are much longer — this metric deliberately captures only time-to-first-output, which is what a user perceives. Cloud stages report a network-bound proxy measured on a 10-sample streaming run (e.g. ElevenLabs v3 TTS TTFA ≈ 0.55s for English); non-streaming TTS reports its total generation time as TTFA, since no audio exists earlier.
Source: run_e2e_eval.py · models/asr/elevenlabs_scribe/model/model.py