Real-Time AI Fails Quietly

I was testing an unrelated change when I opened our embedded voice assistant in a browser and refused the microphone prompt. I expected it to switch to typing. Instead, the whole session closed.

Typing was a supported fallback. There was no reason for the conversation to end just because the browser couldn't reach a microphone. The interesting part was, nothing on our side recorded a failure.

What actually happened was small. The browser reported the refusal with an error type our code didn't expect, so a generic error handler treated it as a failed connection and tore everything down.

Every service involved was healthy. Every request that had been made had succeeded. The user still had a broken experience.

Chat interfaces can fail quietly too. A reply gets cut off, state goes stale, or a stream ends early. Voice and avatar systems add another layer. Audio can go silent, an avatar can stop responding, or a conversation can end unexpectedly without an obvious error.

These failures are particularly easy to miss in real-time systems, where an interaction can continue in a degraded state.

And that's a different kind of reliability problem.

A system can be healthy and still be broken

After that session, I started noticing the same pattern. The logs looked normal, but the interaction wasn't.

A running connection doesn't guarantee useful output. Audio can keep streaming without producing sound. An avatar can render every frame while its expression remains unchanged. The system may be active, but parts of the interaction have already stopped working.

Fallbacks can hide failures. We build them on purpose, like typing when there's no microphone or simpler animation when a component is slow. That's the whole point of having them. The problem starts when a fallback runs and nothing downstream knows. To everything that reads the result, it looks like success.

Users don't report degraded experiences. They leave. In a chat product, a confused user might retry, which at least leaves a trace. Someone watching a frozen face or hearing silence just closes the tab. In the logs, that looks exactly like a conversation that ended normally.

Where real-time interactions break

The microphone bug was one version of this. The other three each broke in a different place.

The launcher: the start never happened. After the assistant moved to its own subdomain, it only accepted its start message from that subdomain. The message sent by the page hosting it was dropped. The panel opened, the face appeared, and nothing happened. No request failed because the interaction never started. It only showed up when we tested the built site, not a local setup.

The narration: completion is not correctness. One pipeline produced narrated videos. A speech dependency started rejecting requests. The pipeline had a fallback for missing audio, which produced silence, and the job was marked complete.

From the pipeline's point of view, everything worked. Every step ran and the output existed. What the user received was a finished video with no voice. And because complete meant final, there was no way to redo it.

What should have caught it was a check on the output itself: is there audible sound where the narration should be? A silent result should have been marked degraded and left open for a redo, not filed as done.

The animation channel: moving is not the same as being seen. We already had a check to detect channels that stopped changing during a session. In one session, the avatar's smile was barely visible, and the check still passed. The channel had technically moved, just not enough to produce a meaningful visual result. A check that asks whether a value changed cannot tell us whether the animation was actually noticeable.

So what should we actually monitor?

Health checks answer "is each part running?" For a real-time system, the better question is "what did the person on the other end see and hear?" That can only be answered where the person is: in the client, at the moment the output is played or drawn.

In practice, it comes down to five questions.

• Did playback actually start? Measure from the end of the user's turn to the first audible output at the client's playback boundary. Audio received and audio played are different events, so track both. The client can't know that a person heard it, but it can know whether sound came out.
• Did it move meaningfully? Track the range each animation channel covers in a session and flag any that never changed. There's a difference between a channel receiving updates, a channel changing meaningfully, and that change showing up on screen. The smile is a good example: one brief change was enough to pass the check, even though the expression barely showed.
• Which mode ran? Text input when there's no microphone is exactly what should happen. What matters is that the system knows it took that path, and for how long.
• Why did it end? A user leaving, an intentional handoff, a connection failure, a timeout and a failed start are five different endings. "Session ended" on its own hides all of them.
• Did the interaction recover? Detecting a problem isn't enough if the user is still stranded. Did the system retry, switch input mode, tell the user what happened, and stay usable? In the microphone case, the right outcome wasn't a better error log. It was the user landing in typing mode.

These checks report numbers, not content. A session summary carries event counts, timestamps and channel ranges, never what was said. We treat "no conversation content in telemetry" as a tested guarantee, not a convention. That's what made it safe to leave the measurement on everywhere, which is where quiet failures happen.

Designing for degraded states

Most of what changed after these incidents was how we treat the paths we don't expect to take.

Fallbacks should announce themselves. When a backup path runs, it should emit an event saying so. That doesn't mean every fallback needs to trigger an alert. The fallback still protects the user. It just stops hiding.

Degraded output shouldn't automatically become final. A fallback can be acceptable in one situation and completely inadequate in another. Text input instead of a microphone is a valid alternative. A video without narration, when narration is expected, isn't. For our narration pipeline, this meant keeping silent or fallback-generated videos open for another attempt before approval. The important part is having clear criteria for what counts as an acceptable result.

Only end the session on errors you mean. A microphone permission refusal doesn't make a conversation impossible, but a genuine connection failure might. Error handling should distinguish between conditions that allow the interaction to continue, those that require switching modes, and those that genuinely require ending the session. A catch-all handler can turn a recoverable problem into a much bigger failure.

Test where the user is. The launcher only broke once the assistant ran on its own subdomain, inside the real host page. Tests that only cover the development setup miss browser permissions, origin differences and embedding behaviour.

These aren't new reliability principles. The difference is, in a live system, a fallback can save an interaction, a retry can save an output, and the right error handler can keep a conversation alive.

The test I'd run first

If you run a voice or avatar system, try deliberately breaking a few things. Refuse the microphone, block one dependency, and watch a full session from the user's side. Then check your monitoring. Does it agree with what you saw about what happened, which mode ran and why the session ended?

In our case the honest answer, more than once, was no. Most of what we've built since is about getting the system to notice that on its own.