Barge-in without the whiplash: Korean turn-taking in a production voice agent

Interruption handling is where a voice agent feels human or feels broken, and there is very little middle ground. Stop too eagerly and every murmur of agreement kills the agent mid-sentence; stop too slowly and the agent talks over the caller. For Korean conversations — and especially for the patient, unhurried speech of elder care — the defaults baked into general-purpose voice stacks get this wrong in both directions.

Four questions, not one

Under the noise, every voice system that handles interruption well is really answering four questions in sequence. Detection: is the person speaking, and have they stopped? Decision: is this a real interruption, a backchannel, or just a mid-thought pause? Execution: given the answer, does the agent keep talking, quiet down, or stop outright? Recovery: once it has yielded, how does it pick the thread back up without making the caller repeat themselves? Almost every failure people actually notice is a wrong answer at the decision stage — the agent mistaking 'go on' for 'stop.' Treating the four as separate problems is what keeps a fix to one from quietly breaking another.

The backchannel trap

The clearest failure is the backchannel. Korean speakers signal that they are listening with short tokens — 네, 응, 어 — that are functionally 'mm-hm', not 'stop talking'. A naive pipeline treats any incoming speech as a barge-in and yields the floor, which turns a supportive listener into a conversation killer. The person is trying to say 'go on', and the agent hears 'I'll take it from here'. The rule of thumb the field has converged on is simple to state and hard to implement: a very short utterance carrying no real content word is a continuer, not a command.

Getting this right is not a matter of a single threshold. It requires separating two questions that stacks often conflate: when has the user started a turn, and when have they finished one. We tune those axes independently, so that starting to speak and finishing a thought are handled by different logic rather than one blunt voice-activity timer.

The ghost in the microphone

Before any of that can work, the agent has to not interrupt itself. When the agent's own voice leaks back into the microphone, a naive pipeline hears speech and dutifully yields the floor to a caller who never said anything — a ghost barge-in, the agent startled by the echo of its own words. Echo cancellation is the unglamorous prerequisite that makes every downstream signal trustworthy, and it takes a moment to lock on at the start of each turn. That is exactly what the grace window right after the agent starts speaking is for: it holds the floor through the vulnerable moment before the caller's echo cancellation settles, so the agent is not fooled by its own voice.

What we tuned

We began by compiling interruption research across the major open-source and commercial voice stacks — the universal mechanisms, where each system draws its lines, and where the academic and practitioner communities disagree. That survey was itself a factory job: several research agents worked the question in parallel, and every finding was tagged by how well it was supported, from a vendor's own documentation down to a community anecdote, so we could tell settled behavior apart from folklore before we changed anything. Then we tuned our own Pipecat-based pipeline against how people actually speak.

Several decisions do the heavy lifting. A patient voice-activity stop window keeps the agent from clipping a caller who simply paused. A word gate means the agent only treats speech as a real interruption once it carries enough content to be one — a single decisive Korean word like 잠깐 or 그만 is enough to stop, but an idle syllable is not. We require a finalized transcript before yielding, which blunts two problems at once: the agent interrupting itself on the echo of its own voice, and a lone backchannel registering as a command. A grace window right after the agent starts speaking covers the moment before the caller's echo cancellation locks on.

On top of that sit two optional models. A backchannel-aware interruption-prediction step can distinguish a genuine barge-in from 네/응 before a turn is even claimed, and it degrades gracefully to the simpler path when it is not available. A neural end-of-turn model runs locally to judge whether a speaker has actually finished, and an optional turn-completion check reads the transcript and issues its verdict in Korean — so someone who trails off mid-thought is gently re-prompted instead of talked over.

Patience is prediction

There is a deeper reason patience pays off. Studies of ordinary conversation across many languages find that people hand the floor back and forth with almost no gap — faster than it takes to plan a reply from scratch — which means humans do not really wait for a turn to end so much as anticipate when it will. A voice agent that only reacts to silence will always feel a beat behind. The point of a turn-completion judge and a backchannel-aware predictor is precisely that they let the system read where a turn is going, not just notice after the fact that it has stopped.

Elder speech has its own rhythm

General voice AI is tuned for brisk, transactional exchanges. Care conversations are not that. People read a phone number back one digit at a time, with pauses between them. They think out loud. They restart sentences. A pipeline that assumes a pause means 'your turn' will chop these moments to pieces.

So we widen the stop window during the situations where patience matters most, letting a caller finish spelling something out or work through a number without being cut off. The governing principle, shared across the whole design, is that a false interruption is worse than a missed one: it is more jarring to be cut off than to wait an extra beat. Our defaults lean conservative on purpose.

Orchestration, not model ownership

None of this required owning a speech model. The recognizer and the synthesizer are best-of-breed components; the differentiation lives in the orchestration layer — which signals to trust, when to trust them, and in which language. That layer is ours, it is tuned for Korean first rather than adapted from English, and it is instrumented on real calls rather than tuned once against a demo.

Turn-taking, it turns out, is a systems problem more than a modeling one, and it is exactly the kind of problem the Vulcan System is built to keep iterating on. Every one of the choices above is a flag we can move, a strategy we can swap, and a behavior we can watch on real conversations — so the pipeline keeps getting better at the one thing that decides whether a voice agent feels like a listener or an interrupter.