About 2,000 agent-to-agent voice simulations, roughly 1,700 of them on EVA. Both sides were real Gemini Live sessions on a live audio line.
01 / Controlled test
Only the harness changed. Task completion rose.
54% to 67% to 80%
Same model, same runtime, same caller, same backend, 10 airline scenarios. EVA's own one-prompt agent completed 15 of 28 calls. Three runs of our flows completed 67% to 80%. We treat 67% as the harness-only floor.
02 / Policy
The model knew the rule. The harness enforced it.
0 of 3 to 8 of 8
A caller changes a flight by choice, then asks for a hotel voucher. Policy allows a voucher only for an airline disruption. The one-prompt agent gave the voucher every time. Our flows refused every time. This is the failure that costs a real airline money.
03 / Where it matters
Dense policy and many tools is where structure pays most
0.89 vs 0.49 plain, airline EVA-A
Airline holds EVA's densest policy: fees, fare differences, standby, refunds, vouchers. Every published airline row is at 0.69 or below. On IT the gain over plain Gemini is small and inside the noise. Most production voice agents look like airline.
04 / Where failures happen
Failures live in the seams, not the model's knowledge
- Ending the audio stream right after a reply: 22% to 46% of next turns lost their first word. With 800 ms of silence first: 2% to 6%.
- Fast tool results cut the reply announcing them: 28% to 31% of replies cut in the first 0.6 s. A 5 s hold took that to 6.5%.
- Post-call text without a frame: 26 of 56 finished. With one framing sentence: 10 of 10.
05 / What fixes it
Move rules out of the model's memory, into places they are enforced
- One objective per stage, with a tool fence: the runtime refuses tools not allowed in that step and says why.
- Typed plugins validate before a call counts. A 5 character confirmation code is rejected with a readback prompt.
- The integration does the arithmetic. The model talks.
06 / The trade
Gemini Live already leads on conversation
0.88 plain Gemini EVA-X, we are at 0.60
Plain Gemini 3.8 Live leads EVA-X, the experience score, on these two domains. The harness bought accuracy and paid for it in smoothness. Closing that gap, with accuracy held, is the work in progress now.