Speech and voice AI systems turn waveforms into text, text into spoken audio, and overlapping talk into speaker-aware streams. This guide owns automatic speech recognition (ASR) and text-to-speech (TTS) pipelines, speech-specific metrics, domain shift in audio, diarization, streaming constraints, on-device versus cloud placement, and voice biometric consent risks. It is a modality guide under deep learning, sibling to computer vision and multimodal AI—not a generic NLP essay, not a full large-language-model textbook, and not a dialog/CX product architecture (that altitude belongs with conversational systems once published). Generative speech synthesis patterns also touch generative AI; keep synthesis controllability and spoofing risks here.
Build speech products around signal conditions and evaluation, not around demo transcripts recorded in quiet rooms.
Speech signals as a modality
Speech is a time-varying acoustic signal shaped by physiology, language, room acoustics, codecs, and devices. Sampling rate, bit depth, compression (telephony AMR, VoIP Opus, mobile AAC), and packet loss change what models hear. A system trained on studio podcasts will not automatically transfer to call centers or factory floors.
Unlike static images, speech is streaming by nature: latency budgets, partial hypotheses, and endpointing matter as much as final accuracy. Unlike plain text NLP, errors are often acoustic (noise, accent, channel) before they are linguistic.
Multimodal stacks may fuse speech with vision or text; fusion ownership stays with multimodal AI. This page owns the audio path: representation, recognition, synthesis, and audio-domain failure modes.
Automatic speech recognition pipelines
Classical ASR stacked acoustic models, pronunciation lexicons, and language models. Modern end-to-end systems map audio features to text with encoder–decoder or transducer architectures, often still aided by external language models or biasing lists for rare terms.
A production pipeline includes: capture and codec handling, voice activity detection, feature extraction or learned front-ends, the recognition model, decoding (beam search, timestamping), inverse text normalization (numbers, dates), punctuation/capitalization recovery, and optional domain biasing.
Custom vocabulary and proper-noun boosting are product features, not afterthoughts. Healthcare, legal, and brand names fail first when biasing is absent. Measure those slices explicitly.
| Stage | Job | Typical failure |
|---|---|---|
| Capture / codec | Preserve usable signal | 8 kHz telephony mismatch |
| VAD / endpointing | Find speech regions | Cutoffs; babble false triggers |
| ASR core | Audio → tokens/text | Domain/accent shift |
| Decoding / ITN | Readable transcript | Broken numbers/entities |
| Biasing | Domain terms | Over-bias hallucinations |
Word error rate and domain mismatch
Word error rate (WER) counts substitutions, deletions, and insertions against a reference. It is necessary and incomplete. Named-entity errors that wreck downstream workflows may barely move WER. Conversely, harmless filler differences can inflate WER without hurting users.
Always report WER (or character error rate) on your domain audio: accents, codecs, SNR bands, and speaking styles you ship. Public benchmark wins transfer poorly. Pair WER with entity F1, intent-critical phrase accuracy, and human transcript preference on hard slices.
Domain mismatch is the default, not the exception. Adaptation via fine-tuning, adapters, or biasing helps when you have matched labeled audio; synthetic speech augmentation helps only when acoustic fidelity is validated.
Streaming versus batch recognition
Batch ASR can wait for the full utterance and usually yields better accuracy. Streaming ASR emits partial results under latency SLOs for dictation, captioning, and voice UIs. Transducer and streaming encoder designs trade look-ahead for speed.
Endpointing errors create clipped words or awkward pauses. Partial hypothesis instability (flickering transcripts) harms UX even when final WER is fine. Evaluate partial stability and time-to-final separately from batch WER.
For analytics pipelines, batch is often enough. For conversational turn-taking, streaming quality dominates. Do not buy a batch model and wrap it in a fake stream without measuring partials.
Text-to-speech and prosody control
TTS maps text (plus optional style controls) to waveforms. Modern neural vocoders and end-to-end speech generators improve naturalness while introducing new risks: speaker cloning, inconsistent prosody, and brittle pronunciation of rare tokens.
Controllability matters for products: speaking rate, emphasis, emotion, pauses, SSML-like markup, and lexicon overrides for brands. Mean opinion score (MOS) and side-by-side preference tests beat WER-style thinking. Intelligibility in noise and on small speakers can diverge from headphone MOS.
Voice identity is a biometric-adjacent asset. Consent, watermarking where available, and abuse monitoring belong in the synthesis launch checklist—not only in a marketing “natural voices” claim.
Speaker diarization and overlapping talk
Diarization answers “who spoke when.” Pipelines typically detect speech, extract speaker embeddings, cluster or assign speakers, and align with ASR. Cloud ASR/TTS APIs from vendors such as OpenAI still need the same eval discipline. Overlap (simultaneous talk) remains hard; many systems under-report overlap regions.
Meeting products fail when diarization swaps speakers during crosstalk or when short turns are absorbed into the wrong cluster. Evaluate diarization error rate (DER) on your meeting acoustics and microphone topologies (single far-field vs headset mix).
Diarization is not the same as voice authentication. Clustering speakers in a meeting does not prove legal identity.
Voice biometrics and consent
Speaker verification and identification use voice as a biometric signal. False accept and false reject rates, presentation attack detection (replay, synthesis), and enrollment quality dominate. Treat voiceprints as sensitive personal data with retention limits and explicit consent.
Do not silently enroll users from call recordings for authentication without policy review. Product copy that says “voice ID” without spoofing resistance is incomplete security.
Separate product goals: convenience unlock versus forensic identification. The latter needs higher assurance and different governance.
Noise, accents, and code-switching failures
Additive noise, reverberation, competing talkers, and music beds destroy ASR that looked strong on clean sets. Accents and dialects inside the same official language are domain shifts. Code-switching (mixing languages mid-utterance) breaks monolingual assumptions in both ASR and TTS lexicons.
Mitigations include matched training data, robust front-ends, multilingual models, and per-locale evaluation gates. “One global English model” is a procurement fantasy for global call centers.
Accessibility stakes are high: users with speech differences, dysarthria, or assistive devices are often underserved. Include those cohorts in eval plans when the product claims broad voice access.
On-device versus cloud speech stacks
On-device ASR/TTS improves latency, offline availability, and privacy; it constrains model size and accuracy. Cloud stacks unlock larger models and continuous updates at the cost of uplink audio, residency questions, and outage dependence.
Hybrid designs run wake-word and early partials on device, then escalate to cloud for final hypotheses or complex languages. Measure energy, thermal throttling, and chipset variance—placement themes that also appear in edge deployments without turning this into a generic infrastructure guide.
Regulated audio (health, finance, government) may require residency, encryption in transit and at rest, and retention controls for both raw audio and transcripts. Transcripts inherit sensitive content even when audio is deleted.
Evaluation beyond WER for product quality
Product eval should include: task success (did the form fill correctly?), entity accuracy, punctuation usefulness for reading, caption readability, TTS intelligibility in target devices, barge-in responsiveness, and escalate-to-human rates for voice bots.
Human review remains necessary for ambiguous audio. Automatic metrics miss semantic near-misses (“fifteen” vs “fifty”) that wreck money and medicine flows. Build golden audio sets with licensed rights to re-use.
When speech feeds an LLM or agent tool loop, measure the combined system—but keep ASR/TTS quality owned as speech metrics so you can tell whether failures are acoustic or dialog policy. Agent orchestration depth stays with AI agents; dialog CX depth with conversational AI when live.
Safety: spoofing and deepfake audio
Synthetic speech can impersonate speakers, bypass naive voice authentication, and scale social engineering. Defenses include liveness/presentation attack detection, multi-factor authentication instead of voice-only unlocks, provenance watermarking where supported, and media authenticity workflows for publishers.
Detection models drift as generators improve. Do not treat a single detector score as permanent proof. Pair technical controls with process: call-backs on known channels for high-risk actions.
Abuse monitoring for TTS APIs (cloning celebrity voices, fraud scripts) is an operator responsibility. Rate limits and use-case contracts matter as much as model quality.
How this guide differs from CV, multimodal, and conversational AI
Computer vision owns spatial visual tasks. Multimodal AI owns cross-modal fusion and shared representations. Conversational AI owns dialog management, CX flows, and turn policies. Speech AI owns the audio transducers and their metrics. Generative AI may frame speech synthesis as a modality; evaluation of prosody, cloning risk, and ASR domain shift remains here.
If a section becomes a general LLM prompting guide or a contact-center playbook without acoustic substance, it has left this page’s ownership.
Worked sketches
Call-center analytics: batch ASR on 8 kHz audio; custom entity lists; WER + entity F1; redact transcripts; diarization only if speaker roles matter.
Mobile dictation: streaming ASR with partial stability SLO; on-device option for privacy; personalize rare names with user lexicons.
IVR replacement voice bot: streaming ASR + TTS; measure task completion and barge-in; keep dialog policy outside pure speech metrics.
Meeting notes: far-field array ASR + diarization; overlap stress tests; do not claim identity verification from diarization labels.
Operational checklist
Codec and sample-rate matrix documented. Domain audio golden set licensed. WER plus task/entity metrics owned. Streaming partial SLOs defined if live. TTS consent and abuse controls reviewed. Spoofing plan for any voice biometric. Residency and retention for audio/transcripts approved. On-device vs cloud decision recorded.
Acoustic front-ends and channel normalization
Before the recognizer, systems still fight channels. Hand-crafted filterbanks, learned front-ends, dereverberation, and beamforming on microphone arrays change the effective signal. Far-field smart speakers and headset mics are different products wearing the same ASR brand name.
Channel normalization and speaker adaptation (including i-vector or embedding-conditioned models) reduce mismatch when enrollment audio exists. Without matched channels in the training mix, adaptation becomes a band-aid on a broken domain plan. Log device and app versions beside transcripts so regressions can be traced to capture changes, not only model weights.
Packet loss concealment and jitter buffers alter timing. Streaming ASR that ignores network artifacts will look fine on Ethernet tests and fail on mobile networks. Include lossy uplink traces in the golden set.
Wake words, duty cycles, and always-listening design
Wake-word detectors are small always-on models with false-accept economics measured in privacy incidents per million hours. Tune thresholds per locale and acoustic environment. A living-room TV false trigger is not the same as a warehouse false trigger.
After wake, the system opens a listening window. Closing that window too early clips commands; leaving it open too long captures private speech into cloud buffers. Document what audio leaves the device, for how long, and under which consent banner. On-device wake with cloud ASR is a common hybrid—make the handoff explicit in architecture reviews.
Duty-cycle energy budgets on battery devices constrain which models can run continuously. Thermal throttling mid-utterance creates mysterious WER spikes that look like “model quality” in dashboards.
Pronunciation lexicons and grapheme-to-phoneme
TTS and ASR both need pronunciation control. Grapheme-to-phoneme (G2P) models guess phone sequences for novel words; lexicons lock brand and clinical terms. Wrong stress patterns destroy TTS trust even when the waveform is “natural.” Wrong pronunciations in ASR references poison training and eval.
Maintain a pronunciation change process: request, linguist or power-user review, locale tagging, and regression clips. Crowdsourced pronunciations without review create inconsistent voices across releases.
Homographs (“read” present vs past) and heteronyms need context-aware G2P or SSML hints. Pure end-to-end models still benefit from explicit overrides for high-stakes tokens.
Multilingual routing and language identification
Language identification (LID) routes audio to the right recognizer or to a multilingual model. LID errors cause cascading ASR failures that look like “bad accuracy” for a language the system never selected. Evaluate LID on code-switched and short utterances—the hard cases users actually produce.
Product policy must define fallbacks: ask the user to confirm language, use a multilingual model, or escalate to a human. Silent wrong-language decoding wastes the entire turn.
Locale is not language alone: en-IN, en-GB, and en-US differ in ITN, spelling, and acoustic priors. Ship locale packs as versioned artifacts beside model IDs.
Redaction, logging, and transcript hygiene
Transcripts contain payment card numbers, health details, and credentials spoken aloud. Automatic redaction (pattern + ASR-aware NER) should run before durable storage and before vendor analytics tools. Redaction failures are privacy incidents even when audio was encrypted in transit.
Decide whether raw audio is retained at all. Many products need transcripts only; keeping audio “just in case” expands breach blast radius and biometric exposure. If audio is retained for dispute resolution, define TTL, access roles, and legal hold exceptions.
Debug logs that store partial hypotheses can leak more than final transcripts because users correct themselves mid-sentence. Apply the same retention class to partials.
Closing
Speech AI succeeds when acoustic reality, streaming constraints, and speech-specific evaluation drive design. Own ASR/TTS pipelines, diarization, and voice risk here; defer dialog product architecture and cross-modal fusion to their guides. Quiet-room demos are not domain proof.
Feature biasing and language models in ASR
External language models and fused shallow fusion can repair fluent phrasing while inventing plausible wrong words. Biasing lists raise recall for brands and drugs and can over-trigger similar-sounding tokens. Tune biasing strength on a held-out entity set; monitor hallucination-like substitutions after each lexicon change.
Contextual biasing from screen state (the form field the user is filling) often beats giant generic LMs for product accuracy. Keep PII out of logs when contextual text is sensitive.
Punctuation, capitalization, and inverse text normalization
Raw token streams are not user-ready. Punctuation models, truecasing, and ITN turn “twenty five dollars” into “$25.” Errors here look like ASR failures to users even when the acoustic model was fine. Evaluate readable transcript quality separately from raw WER on tokenized references.
Locale rules differ: dates, currencies, and honorifics need per-locale tests. A US-centric ITN layer on global audio creates silent product bugs.
Latency budgets and barge-in
Voice interfaces need tight loops: capture → partial ASR → policy → TTS playout. Barge-in (user interrupts TTS) requires stopping synthesis quickly and re-listening without cutting the user’s first syllables. Measure time-to-first-partial, time-to-final, and TTS stop latency as product SLOs.
Buffer bloat and client-side audio queues cause “robot talking over user” incidents that no WER dashboard explains. Instrument the full media path.
Data rights for audio corpora
Training and eval audio carry voice biometrics and ambient privacy (other people in the room). Prefer licensed datasets, customer contracts that allow model improvement only when explicit, and synthetic data when rights-safe and acoustically validated. Do not scrape phone calls into training by default.
Vendor subprocessors that store audio for “quality” need the same scrutiny as primary stores. Deletion pipelines must cover audio objects and derived embeddings.
Anti-patterns
Buying ASR on public WER alone. Ignoring telephony codecs. Voice-only authentication without spoofing resistance. Treating diarization labels as legal identity. Shipping TTS cloning without consent tooling. Evaluating only final hypotheses for streaming UX. Assuming multilingual marketing copy means code-switch robustness.
Team interfaces
Speech ML owns models and audio metrics. Client/media engineers own capture, codecs, and barge-in. Product owns task success definitions. Security/privacy owns biometric and retention policy. CX/dialog designers own conversation policy—coordinate, do not collapse into one “voice AI” backlog without owners.
Write a speech system charter: languages/locales, codec matrix, online vs batch, biometric or not, retention, and golden-set owner.
Where speech AI sits in the Knowledge graph
Parent systems context sits in deep learning. Visual sibling: computer vision. Fusion sibling: multimodal AI. Generative framing: generative AI. Tooling loops: AI agents (brief). Dialog CX: conversational AI after it publishes in this batch. Keep acoustic ownership here so neighboring pages can link without rewriting ASR.
Offline decoding, lattices, and confidence
Batch systems can afford richer search: lattices, confusion networks, and rescoring passes. Confidence estimation—whether token-level or utterance-level—drives human review routing and active learning. Poorly calibrated confidence sends the wrong calls to agents and starves retraining of useful errors.
Use confidence as a decision feature with its own validation, similar in spirit to probability calibration in supervised learning, but estimated under acoustic uncertainty. Never treat a single model-internal softmax as a universal confidence without slice checks on noisy and accented audio.
Forced alignment tools map transcripts back to time for captions and training. Alignment errors create bad supervision for punctuation models and speaker-attributed captions. Spot-check alignments on music beds and overlapping speech.
TTS voice design and brand continuity
Product voices are brand assets. Consistency across devices, rates, and SSML features matters more than winning a one-off MOS bake-off with a voice you cannot operationalize. Maintain a voice style guide: allowed emotions, prohibited styles, pronunciation owners, and update cadence.
Multi-speaker TTS catalogs need governance: which personas are approved, which languages they support, and how cloning requests are reviewed. Deprecating a voice is a product migration—offer timelines and fallbacks like any other API break.
Expressive controls that overfit to demo scripts often collapse on long-form help articles. Evaluate paragraph-level listening tests, not only single-sentence MOS.
Captioning, accessibility, and live media
Live captions for broadcasts and classrooms combine streaming ASR, punctuation, and display constraints (line length, reading speed). Latency and stability dominate; a slightly higher WER with stable partials can beat a flickering lower-WER stream.
Accessibility requirements may mandate caption quality levels, speaker labels, and sound event cues. Diarization and non-speech event tagging become accessibility features, not niceties. Test with Deaf and hard-of-hearing reviewers on real content types—not only clean podcasts.
Translated captions add a machine-translation stage after ASR. Attribute errors correctly: acoustic vs translation. Mixing them into one “caption WER” hides the failing stage.