Reliability harness on Gemini Live

Production reliability for voice and multimodal agents. Built on Gemini Live.

Great demos on the Gemini Live API are possible today. We wanted to go further and make those agents reliable in production, because that is where deals stall: between pilot and production.

Before building anything else, we needed to see what the agent was saying and doing before our customers did. So we built Lens first. Everything else in the harness grew from what Lens showed us.

Inside the voice server

One supervisor per call controls the live model.

Audio flows straight between the caller and Gemini Live, so latency stays low. The supervisor sits beside that path. It hears every event, decides what is allowed, and controls playback, model input and every tool call.

Dinodial voice server harness, one live call Audio flows Caller to Transport to Partner to Gemini Live. The Supervisor is outside the audio path and controls it through events. Tool calls go from Gemini through Partner to the Supervisor, which checks the stage and sends allowed calls to Integrations, where they are validated and executed in a sandbox. The result returns to Gemini. Command starts and stops the call and receives the call record. caller audio + speech start / end direct audio path. The supervisor is not in it. WebSocket Caller Phone carrier or web widget TRANSPORT Hears the caller Audio in and out, our own VAD, marks when audio has played PARTNER Runs the model session Tracks each reply, opens and closes model input on command Gemini Live Speech to speech AI Studio or Vertex speech start / end, audio played stop playback, prompt reply started / done, interrupted, tool call open / close input, tool reply SUPERVISOR · ONE PER CALL Decides what happens next, one event at a time 1 Listen next event or timer 2 Decide stage rules, tool policy 3 Act one effect at a time Flow runtime: current stage, which tools are allowed, where each outcome goes next. Timers: silent caller, call deadline, goodbye and post call steps. COMMAND Control channel Starts and stops the call. Receives the call record when it ends (Lens, Wallet). gRPC INTEGRATIONS Sandboxed tools Validate quick check, returns verdict Execute own sandbox, returns result 1 2 3 4 5 live audio control events 1 tool call path, in order
A tool call, step by step
  1. 1 Gemini asks for a tool. The Partner passes the request to the Supervisor.
  2. 2 The Supervisor checks the current stage: is this tool allowed here, now, this many times? If not, it is rejected and nothing runs.
  3. 3 Allowed calls go to Integrations. The input is validated in one sandbox, then the work runs in a separate one.
  4. 4 A verdict comes back right away, then the result.
  5. 5 The Supervisor sends the tool reply through the Partner to Gemini, and the conversation continues.
An interruption, step by step
  1. 1 Transport hears the caller start talking (our VAD) and tells the Supervisor.
  2. 2 The model also stops its reply, and the Partner reports the interruption.
  3. 3 If a reply is playing and that line is not protected, the Supervisor tells Transport to stop playback at once.
  4. 4 A reply only counts as finished when the caller has actually heard it.
  • Interruptions handled in the server. When the caller talks over the agent, playback stops right away unless that line is protected.
  • The model cannot skip steps. A tool that is not allowed in the current stage is rejected before it runs.
  • Every call ends cleanly. Timers cover silent callers, long calls and goodbyes.
  • Model swappable. The same supervisor runs on Gemini on AI Studio or Vertex AI, OpenAI or xAI.

The tools around the harness

bench.dinodial.ai

What we learned benchmarking Gemini Live

Same model, with and without the harness. We ran Gemini Live through ServiceNow's public EVA voice benchmark inside our harness and recorded every call.

EVA-A accuracy pass rate · airline + IT
Gemini Live, plain (EVA board)0.52
With extended thinking (EVA board)0.61
Inside the Dinodial harness0.75

Our run used a random panel of 24 scenarios and our own simulated caller. Caveats on the bench page.

Use-case map

Where we are taking the harness. Voice today, voice plus vision next.

The same harness runs every agent below: Lens, Supervisor, Sandbox, Simulation and the CLI. Left to right adds vision. Top to bottom shows how close each one is to production. Click any point for the detail.

Voice in productionVoice only. Live or piloting
Multimodal, demo readyVoice + vision. Live or piloting
Voice, next upVoice only. Exploring
Multimodal frontierVoice + vision. Exploring
LivePilotExploring

Vertical position: Live (top), Pilot (middle), Exploring (bottom).