Production reliability for voice and multimodal agents. Built on Gemini Live.
Great demos on the Gemini Live API are possible today. We wanted to go further and make those agents reliable in production, because that is where deals stall: between pilot and production.
Before building anything else, we needed to see what the agent was saying and doing before our customers did. So we built Lens first. Everything else in the harness grew from what Lens showed us.
One supervisor per call controls the live model.
Audio flows straight between the caller and Gemini Live, so latency stays low. The supervisor sits beside that path. It hears every event, decides what is allowed, and controls playback, model input and every tool call.
- 1 Gemini asks for a tool. The Partner passes the request to the Supervisor.
- 2 The Supervisor checks the current stage: is this tool allowed here, now, this many times? If not, it is rejected and nothing runs.
- 3 Allowed calls go to Integrations. The input is validated in one sandbox, then the work runs in a separate one.
- 4 A verdict comes back right away, then the result.
- 5 The Supervisor sends the tool reply through the Partner to Gemini, and the conversation continues.
- 1 Transport hears the caller start talking (our VAD) and tells the Supervisor.
- 2 The model also stops its reply, and the Partner reports the interruption.
- 3 If a reply is playing and that line is not protected, the Supervisor tells Transport to stop playback at once.
- 4 A reply only counts as finished when the caller has actually heard it.
- Interruptions handled in the server. When the caller talks over the agent, playback stops right away unless that line is protected.
- The model cannot skip steps. A tool that is not allowed in the current stage is rejected before it runs.
- Every call ends cleanly. Timers cover silent callers, long calls and goodbyes.
- Model swappable. The same supervisor runs on Gemini on AI Studio or Vertex AI, OpenAI or xAI.
The tools around the harness
- 01
Lens
See every interruption, transcript and tool call, so problems surface before customers notice.
Open → - 02
Supervisor
Keeps the live model on script, inside guardrails, with clean turn taking.
Open → - 03
Sandboxed tools
Every action is isolated and checked, so mistakes stop before your systems.
Open → - 04
Simulation
Voice to voice test callers find failures before launch and shorten time to go live.
Open → - 05
CLI + MCP
Coding agents read calls, propose fixes and ship new versions without a dashboard.
Open →
What we learned benchmarking Gemini Live
Same model, with and without the harness. We ran Gemini Live through ServiceNow's public EVA voice benchmark inside our harness and recorded every call.
Our run used a random panel of 24 scenarios and our own simulated caller. Caveats on the bench page.
Use-case map
Where we are taking the harness. Voice today, voice plus vision next.
The same harness runs every agent below: Lens, Supervisor, Sandbox, Simulation and the CLI. Left to right adds vision. Top to bottom shows how close each one is to production. Click any point for the detail.
Vertical position: Live (top), Pilot (middle), Exploring (bottom).