Over the past few months, I've been building voice prototypes: weekend experiments like a conversational tic-tac-toe game, and more serious explorations, a real-time bidirectional translator and an ever-evolving voice assistant. It's a shift from the text-first GenAI work I wrote about in Building a Gen AI Product. Each project started with the same assumption: voice is just text with audio. Each one broke that assumption in a different way.
What I've learned is that voice isn't an audio layer you bolt onto a chat system. It's a fundamentally different paradigm, one that requires specialized architecture, conversation design, and user experience thinking. The assumption that voice = text + audio leads to systems that feel awkward, perform poorly, and fail to deliver on the promise of natural conversation.
This isn't just a theoretical distinction. The differences cascade through every layer of the stack: from how the model processes input, to how state is managed, to where privacy boundaries fall, to what good UX even feels like.
Fig. 1: Voice conversation vs. text chat: the eight assumptions that break when voice is treated as text with an audio layer, from real-time error correction and sub-second latency budgets to conversational failure recovery, environmental constraints, and voice as biometric data. Image by author.
Error Correction Changes Everything
Turn-taking has no clean setting. The first version of my assistant treated a brief silence as "I'm done," and it cut people off whenever they paused to think; a 700ms gap to recall a name read as the end of the turn. Give the pause more room and the assistant feels sluggish instead. There is no single value that is both responsive and patient, because the system is guessing at something a text box never has to resolve: whether you are finished.
In text chat, users can edit their messages before sending, review conversation history, and carefully craft their responses. Voice conversation operates in real-time without the luxury of editing. Once words leave your mouth, they become part of the conversation record, complete with hesitations, false starts, and corrections.
This creates error patterns that your system must handle. Users will naturally say things like "set a reminder for... no wait, create a calendar event for tomorrow at 3pm." The first time my assistant hit a sentence like that, it had already started generating a response to the first half by the time the second half arrived; it was answering a question I'd already retracted. Your system must parse these corrections, understand the user's intent, and respond appropriately. Unlike text, where typos are the primary error mode, voice systems must handle semantic corrections, topic changes, and conversational repairs.
Failure has to be conversational too. There is no error modal to throw and no loading spinner to hide behind: when the system mishears, or a tool call dies mid-turn, recovery happens inside the conversation, mid-sentence, or it doesn't happen at all. A text app can fail silently and let the user retry. A voice agent that goes quiet has already failed.
VoiceTicTac was supposed to be a single voice agent doing several things at once: a chatty in-character "player" my kids could practice English with, a game-command interpreter for their moves, and a rudimentary voice authentication flow ("tell me your name"). I tried to make one voice-chat LLM hold all three roles with a single prompt. Guess what, it didn't work. The agent kept conflating modes, treating game moves as conversation and conversational asides as game intent. I ended up splitting into two paths: a command path (browser STT → Claude → game state) for moves, and a chat path for the "player" persona. Better prompt engineering at the time would have closed some of the gap. But the deeper pattern I see now, two systems later, is that voice agents asked to hold multiple roles benefit from architectural separation, not just from a sharper system prompt: a "supervisor" that routes intents.
What a Voice LLM Actually Is
A voice LLM is not a general chat model with a speech-to-text front end and a text-to-speech back end glued on. Architecturally, it's a different kind of model, optimized for a different job: fast, streaming audio-in / audio-out transduction under a tight latency budget. It is not optimized for deep multi-step reasoning over a completed prompt.
Three properties distinguish voice-optimized models from general-purpose chat LLMs:
- Streaming, not batched. Voice models begin producing output while the input is still arriving, and revise as more audio comes in. A general LLM takes a complete prompt and emits a complete response. That difference drives a lot of downstream architecture: your business logic has to handle provisional results that may change. And the state you carry is continuous rather than discrete: audio streams, timing patterns, and speaker identity are all maintained simultaneously, not as a tidy list of messages.
- Latency as a first-class constraint. Conversational speech tolerates maybe a few hundred milliseconds of delay before the interaction feels broken. Voice models trade away some reasoning depth to hit that budget. A general LLM that thinks for several seconds is fine in chat and unusable on a call.
- Turn-taking and prosody, not just tokens. Voice models decide when the user has stopped talking, when it's appropriate to speak, and how to modulate tone and pacing. None of this exists in a text model.
You can optimize prompts within this architecture: filler handling and multi-turn ambiguity yield to careful prompt engineering. What you can't do is optimize past the ceiling the architecture imposes. When a failure mode is structural, the fix isn't a better prompt; it's a different stack.
But these architectural properties also set a boundary: a voice model optimized for speed and streaming still relies on its language model to resolve ambiguous audio. When it has never been trained on your domain's vocabulary (drug names, legal terms, company jargon), it will guess wrong. That gap, between general and domain-specific performance, is what the next section is about.
Accuracy and Domain Specialization
Picture a small example. You're transcribing a deposition. The attorney says, "Move that we treat this as a habeas corpus petition." A general recognizer hears that cold and writes down "hey this corpus" or "habit corpus." The acoustic signal is close; the language model is what would resolve it, and a general one has no reason to prefer the legal phrase over more common ones. Now tell that recognizer, before the deposition starts, that today's flow is a constitutional law case and the participants will use terms like habeas corpus, certiorari, and amicus brief. The same audio becomes legible. The recognizer didn't change; its priors did.
Even the best general voice model hits a ceiling the moment your conversation moves into specialized territory. The problem isn't model size, it's that the recognizer has never seen your vocabulary, your accents, your microphones, or your conversational patterns often enough to do better than guess. The numbers bear this out: even models with respectable overall word error rates post markedly higher error rates on clinical entities like drug names and diagnoses, and fine-tuning on domain-specific speech improved medical word error rates by 25 to 34 percent relative in one Interspeech 2024 benchmark. Specialization is what closes that gap.
Concretely, specializing a voice system means some combination of:
- Custom vocabulary and biasing. Telling the recognizer that words like bupropion, Antigravity, or Voulgaris are in-distribution and should be preferred over their more common near-homophones. Without this, proper nouns and jargon get mangled on first contact.
- Acoustic adaptation to accents, noise profiles, and microphones. A model tuned for a conference-room mic in a quiet office will struggle with a Pixel phone in a moving car. Tuning on representative audio is what closes it.
- Domain-tuned language models. A finance-tuned language model writes "GAAP" where a general model hears the word "gap." Same audio, different priors.
- Per-user personalization. The same acoustic model gets better at your voice over time.
- Turn-taking tuning. Customer-support calls have very different pause structures than casual conversation; a model tuned on support audio handles interruptions and handoffs more naturally.
Why bother? Because the accuracy limit is what separates "fun to try" from "relied on daily." A recognizer that hears "Sumatriptan" as "sue Matt rip ten" every fifth call is a pharmacy assistant you can't ship. A voicemail transcriber that loses the caller's name half the time is a feature that erodes trust. A meeting-notes product that can't spell your CEO's name is a demo, not a tool.
The pattern shows up clearly in consumer voice products. Apple's on-device "Hey Siri" trigger adapts to your voice over time, building a speaker profile from your own utterances on top of the same speaker-independent detector everyone gets. That's not a different base model, it's specialization applied per user. Amazon Alexa's recognizer is biased toward contextually relevant vocabulary while it decodes: it boosts the probabilities of words like contact and appliance names, so terms tied to a user's contacts or linked skills survive in the recognition lattice instead of being pruned. YouTube's auto-captions launched in English only and were extended language by language, each new language brought online as Google built out support for it. OpenAI's Whisper (the model much of the consumer ecosystem is built on) shows double-digit accuracy swings across audio conditions and domains.
When I built the bidirectional translator, I learned that voice systems have to model a set of intents that don't meaningfully exist in text chat. The most common one was "qué?" / "what did you say?": a high-frequency repeat-request that text users almost never produce because they can scroll back. The general-purpose stack treated those utterances as new content to translate. I had to add a separate classifier with its own LLM call to distinguish "translate this" from "repeat the previous translation." Same shape with language detection: short utterances voice users actually produce, like "sí", "no", "qué", needed another dedicated layer. The accuracy ceiling wasn't model size; it was that voice surfaces intents text never had to catalogue.
The underlying principle is consistent: voice accuracy is a domain problem, not just a model problem. General models plateau. Specialization on vocabulary, on conversational patterns, on the user's own voice and context, is what gets you from a demo to something people will use every day.
Environmental Constraints That Text Doesn't Face
On the translator, the deployment target was an iPad mounted in a lobby, motion-triggered so a visitor walking up would activate it. The constraints were specific: lobby noise, a 0.5–1 meter speaking distance, two speakers of different first languages, each needing the translation spoken (visitors cannot always read) and on screen (faster for the concierge) at once. Those constraints aren't environment-neutral; they're a design specification the voice layer had to satisfy before any of it was about language.
Voice interfaces face inherent environmental limitations that text interfaces simply don't have. In noisy surroundings, speech recognition accuracy drops significantly, even with high-quality headphones. On the other hand, in quiet settings like libraries or open offices, speaking aloud can be socially inappropriate or disruptive.
These constraints mean voice interfaces cannot universally replace text interfaces. Users require seamless transitions between voice and text based on their environment. Your system must detect these contextual changes and adapt accordingly, maintaining conversational state across different interaction modalities.
While many envision voice interaction like a sci-fi AI companion, the reality is consistently more multimodal than that imagery suggests. The interesting paradigm isn't voice replacing the screen, it's voice using the screen as a complementary input/output device. Voice-first interactions can leverage visual feedback: "Please point to the red area on the diagram," or "Here are the search results. Notice that the ABC metric is lower than expected." And the screen lets the user gesture back into the conversation.
It's simpler to tap and say "fix this" than to describe "this."
The same architectural pattern, viewed from the other side, gives you the fallbacks the environmental constraints demand: captions when audio is off, keyboard input when speaking isn't possible, the agent's ability to highlight on-screen areas. Either direction, the system has to maintain context across both modalities, handle timing synchronization between speech and visual updates, and manage user attention across different interface elements. The future of voice isn't voice-only; it's voice as the primary interaction layer in a multimodal experience.
Voice Data Is More Sensitive Than Text
On the voice assistant, the privacy axis that surfaces earliest is the one in the room, not the one on the server: I use it at the gym. It says things back that the strangers on the next machines can hear, and I say things it can hear too. Some uses survive that (a workout timer, music control); others don't (money, health, anything about anyone else). Before any data leaves the device, the voice channel is itself part of the privacy surface: hands-free computing means the user is, sometimes deliberately and sometimes not, broadcasting into a shared room.
Voice data is fundamentally more sensitive than text. Audio carries biometric identity, emotional state, and environmental context that text simply doesn't. This means privacy in a voice system isn't a feature toggle you flip at the end. It shapes where inference can run, where data lives, how long it persists, and which jurisdictions you're suddenly subject to under GDPR, BIPA, and the emerging wave of AI disclosure rules. The architectural implications run deep enough to deserve their own treatment, which I'll develop more fully in a future piece on voice AI security.
The Parallel vs. Serial Paradox
Voice conversation introduces an interesting paradox around parallel processing. In theory, voice enables parallel interaction: you can talk while doing other tasks, unlike text, which demands visual attention and typing. Voice frees your hands.
But the channel itself is serial in a way visual interfaces are not. Ask a voice agent to "set a timer for four minutes and find a recipe for marinara" and you have already hit the limit: two unrelated tasks, spoken one after the other, because a single stream of words admits nothing else. You can keep ten browser tabs open and switch between them instantly; you cannot maintain ten simultaneous voice conversations without re-establishing context for each one. The channel locks your cognitive thread: one conversation at a time, in order.
Ideally, the voice agent itself would help manage multiple threads, working alongside a visual UI, tracking time, and spotting connections across conversations. That points toward a generation of assistants that goes beyond simple command-response patterns.
Voice Design Is a Distinct UX Discipline
Voice conversation requires design considerations that don't exist in visual interfaces. Tone, pacing, prosody, and emotional context become primary design elements rather than secondary considerations. Conversational hierarchy becomes more important than visual hierarchy. As Smashing Magazine's guide to voice user interfaces puts it, building these interfaces "requires a lot of design expertise in various areas such as conversation design, interaction design, visual and motion design." Voice draws from existing UX practice but demands its own composition.
And role discipline, knowing when the agent engages versus stays out of the way, is a design primitive with no analog in visual UX: a translator that responds to what it should be relaying fails differently than one that mistranslates.
Traditional UX designers must learn conversational design principles, while conversation designers must understand the technical constraints of voice systems. If your team is building voice experiences with only visual interface designers, you're likely missing critical aspects of the conversational experience.
Where 'Voice = Text + Audio' Actually Works
I want to be straight about this because it's tempting to let a thesis swallow up cases it shouldn't.
The simple architecture of STT in, LLM in the middle, TTS out, is genuinely fine for some voice use cases. It's not a stage of evolution that every team needs to graduate from. It's the right answer when:
- The voice is one-way. Audiobooks, accessibility readers, voice notifications, dictation/note-taking tools. There's no conversation. You're either turning audio into text or text into audio; the conversational layer this article is about doesn't exist.
- The interaction is one-shot and stateless. "What's the weather in San Francisco?" → response → done. No follow-ups, no corrections, no turn-taking ambiguity. A request-response pipeline with audio bookends does the job.
- The vocabulary is small and fixed. Smart home commands, IVR menus, in-car controls. When the grammar is constrained enough that ambiguity rarely arises, you don't need turn-taking, mid-utterance correction handling, or domain-tuned language models. You need a tight intent classifier with audio in front of it.
- The latency budget is generous. Asynchronous voice, such as voicemail, voice messages exchanged like texts, and podcast-style interactions, tolerates seconds of delay. The hard constraint that forces voice-specific model architecture (sub-500ms turn-taking) doesn't apply.
The architecture I'm advocating for in this piece is for the case where voice is bidirectional, interactive, and open-ended; where you actually want to talk to the system, not issue it commands. Conversational voice agents. Real-time translation. Voice assistants that hold context across turns. Teams that pick one of those goals and reach for the simple architecture struggle, because that architecture is what worked the last time.
If you're shipping a voice feature that falls into one of the four categories above, this article is interesting but not load-bearing for your stack. If you're shipping a voice agent that needs to converse, it is.
Bolting audio onto a chat architecture is the rational starting point, and for some voice features it's the right one. The moment voice has to converse, though, that architecture stops compounding and starts cascading. This piece is the framework I wish I'd had; the one you actually need will come from your own first failure.
What's Next
There's much more to building voice AI for production. Topics I plan to explore:
- Voice pipeline architectures. A progression from sequential processing to full bidirectional real-time, with guidance on choosing the right level for your use case.
- Engineering challenges. Latency budgets, WebRTC gotchas, interruption handling, and why your infrastructure assumptions from text chat will fail you.
- Voice AI's unique attack surface. Security threats from prompt injection via speech to deepfake voice impersonation.
— George · voulgaris.net