Methodology

• STT & TTS — Google FLEURS test split (English 647, Chinese 945, Malay 749, Tamil 591 clips).
• End-to-end pipeline — VoiceBench (English), a spoken-instruction benchmark with 7 subtasks.
• End-to-end pipeline — SEA-HELM (English, Malay, Tamil, Indonesian, Chinese — coverage varies per subtask), 2 subtasks adapted from the SEA-HELM text benchmark.

Both the reference and the hypothesis are normalized with the same pipeline before any error rate is computed — this removes cosmetic differences (case, punctuation, spacing, script variants) that would otherwise inflate WER/CER with false errors.

Per-language settings in use:

*Tamil uses the Indic normalizer (canonicalizes the script, expands numbers, keeps combining vowel marks). Tamil WER is still inflated by compound-word segmentation — the same sounds written as one word vs. two (e.g. வேகவைக்கப்பட்ட vs. வேக வைக்கப்பட்ட) count as word errors — so the table shows CER alongside WER; externally published Tamil "WER" figures are usually character-level and comparable to our CER.

Edge case: if a string is empty after normalization, the sample scores 0.0 when both ref and hyp are empty, else 1.0 (rather than dividing by zero).

Source: scorers/common/transcription_scorer.py · configs/languages_config.yaml

The STT model produces the transcript during generation; scoring runs no inference. WER and CER are computed with jiwer on the normalized reference vs. hypothesis:

WER = word_error_rate(ref_norm, hyp_norm)  ·  CER = character_error_rate(ref_norm, hyp_norm)

TTFT (time-to-first-token) is reported per model: streaming models (e.g. ElevenLabs scribe_v2_realtime) stamp it at the first partial transcript.

The Overall column is the mean error rate across a model's languages — WER for space-delimited languages (including Tamil), CER for space-free ones (Chinese). Tamil additionally shows its CER in a separate column, since its WER overstates errors on compound segmentation.

Source: scorers/common/transcription_scorer.py · gateway leaderboard/system_board.py (error_rate, overall)

TTS is scored two ways:

• Intelligibility (WER/CER) — the generated audio is re-transcribed by the language's reference STT model (whisper-large-v3), and that transcript is scored against the original TTS input text (same normalization as STT). Low WER/CER means the synthesized speech is clearly recognizable.
• Naturalness (UTMOSv2) — see below.
• TTFA (time-to-first-audio) — how it's obtained depends on the model's serving mode:
  • Streaming local (Qwen-TTS via vLLM) — measured per-clip as the wall-clock to the first audio chunk.
  • Streaming cloud (ElevenLabs v3) — network-bound, so the figure reflects round-trip latency as much as synthesis.
  • Non-streaming (OmniVoice, flow-matching diffusion) — the whole clip is denoised at once, so the first audible audio exists only when the full clip does: TTFA = total generation time, nearly duration-independent.

Source: scorers/common/transcription_scorer.py (task_type="tts") · models/tts/elevenlabs/model/model.py

UTMOSv2 is a reference-free mean-opinion-score predictor: it estimates a naturalness score (~1–5, higher is better) from the audio alone — no reference text or audio needed. The pretrained model is built once and run over the whole directory of generated wavs, returning a predicted MOS per clip that is averaged per (model, language).

Source: scorers/tts/mos_scorer.py (utmosv2.create_model)

RTF is how long the model takes relative to the length of the audio, so it measures throughput over the whole clip, not the wait before the first piece arrives (that is TTFT/TTFA above). Below 1 is faster than real time; 0.25 means a clip is handled in a quarter of the time it takes to play.

STT: RTF = inference_time_s / input_duration_s  ·  TTS: RTF = total_time_s / audio_duration_s

Published as the mean over a (model, language)'s clips; Overall is the mean across the languages a model was tested on. For cloud APIs the measured time includes the network round-trip, so their RTF reflects service latency as much as model speed and is not comparable to a locally served model.

Source: common_utils/asr/base_asr_model.py · common_utils/tts/base_tts_model.py · run_{asr,tts}_eval.py

Finalization latency measures the time to a completetranscript for a cold 5.0-second clip: the leading 5 seconds of a clip are sliced off, transcribed through the model's own entry point, and timed. Published as the median of 10 such clips after one untimed warm-up — the same procedure for every model, which is what makes the numbers comparable.

It is not RTF × 5. Every call pays a fixed cost that does not depend on the clip's length — warm-up, audio decode, network round-trip. RTF divides by that length, so on the corpus's ~13s clips the fixed cost barely shows, while on a 5-second clip it does; RTF × 5 misses the real figure in both directions. For a streaming model this is the total compute to consume 5 seconds in one go, not its live-session latency — a lab measurement, not a production figure.

A run-level probe, so there is one value per model and no per-language variation. Measured in a single later pass on one machine (NVIDIA RTX PRO 6000, or the provider's service for cloud models), whereas the WER and RTF figures beside it come from the original runs, whose hardware was never recorded.

Source: common_utils/asr/fixed_window_latency.py · common_utils/execution_env.py · evaluation/docs/stt_latency_metrics.md

VoiceBench measures task success of the full cascade on spoken instructions. For each sample the LLM is scored on two arms:

• Text (topline) — the LLM answers the gold text prompt (no STT errors).
• Audio (cascade) — the LLM answers the STT transcript, so it inherits any STT mistakes.

The gap between them is the STT front-end's downstream cost on answer quality. Each subtask is scored differently:

On a VoiceBench row of the Voice Agent leaderboard, Acc (text) and Acc (audio) are the macro-average of these per-subtask scores across all subtasks, for the gold arm and cascade arm respectively. Each score is normalized to 0–1 first — for VoiceBench that means dividing the gpt_1to5 judge ratings by 5; every other subtask metric here is already a 0–1 fraction.

Source: scorers/e2e/voicebench_scorer.py · scorers/e2e/external_libraries/voicebench/

SEA-HELM adapts four tasks from the (text-only) SEA-HELM benchmark suite into a voice cascade: each prompt is synthesized to speech, transcribed back, and answered by the LLM — same Text vs. Audio two-arm design as VoiceBench above. Every subtask is scored rule-based (no LLM judge):

Language coverage differs per subtask (a real gap in each task's own upstream data, not a bug) — a language missing a subtask's data simply contributes no samples to that one subtask, while the rest still score normally.

On a SeaHelm row of the Voice Agent leaderboard, Acc (text) and Acc (audio) are the macro-average of whichever subtasks have data for that pipeline/language, for the gold arm and cascade arm respectively. Each score is normalized to 0–1 first — ifeval's language-normalized accuracy is SEA-HELM's own 0–100 convention and is divided by 100, while the other subtask metrics are already 0–1 fractions.

Because the leaderboard rows on (pipeline, language, benchmark), a language covered by both benchmarks gets one VoiceBench row and one SeaHelm row for each pipeline, and a SeaHelm-only language gets a SeaHelm row and no blank VoiceBench one. The two accuracy columns therefore mean whatever that row's Benchmark cell says.

Source: scorers/e2e/seahelm_scorer.py · scorers/e2e/external_libraries/seahelm/

The pipeline latency is the time from the user finishing speaking to the first audio of the reply — the sum of what each stage contributes:

latency = STT finalization latency + LLM TTFT + TTS TTFA

• STT finalization latency — time to a complete transcript for a fixed 5s clip (see above). The cascade waits on the whole transcript, not the first token: the LLM cannot be prompted until the STT stage is done.
• LLM TTFT — time to the first response token on the cascade (audio) arm, averaged across the subtasks of that row's benchmark.
• TTS TTFA — time to the first synthesized audio chunk (streaming).

The STT leg uses finalization latency: the fixed 5s window is produced the same way for every model.

Note: the STT leg is measured on a constant 5-second clip while the LLM and TTS legs are measured on the benchmark's own utterances, so the total is a comparable “what a 5-second turn costs this stack” figure rather than the wall-clock of these specific benchmark items. Full generation times (whole transcript / whole reply / whole clip) are longer still. Cloud stages are network-bound, so their figures include round-trip latency; non-streaming TTS reports its total generation time as TTFA, since no audio exists earlier. A pipeline whose STT model has no finalization latency figure shows no total latency at all.

Source: run_e2e_eval.py · common_utils/asr/fixed_window_latency.py · gateway leaderboard/voice_agent.py (total latency)