Dinodial
← Back to harness
What we learned from bench.dinodial.ai

Same Gemini Live model. Harness on, accuracy up.

We held the model fixed and ran Gemini 3.8 Live on ServiceNow's public EVA voice benchmark, inside the Dinodial harness. On a random panel of 24 airline and IT scenarios it scored 0.75 on EVA-A, the accuracy pass rate. The same model on EVA's published board scores 0.52 plain. Every call is recorded and public.

0.75EVA-A, Dinodial harness + Gemini 3.8 Live, thinking off
0.52EVA-A, plain Gemini 3.8 Live on EVA's board
0.74EVA-A, board leader on these two domains (Grok Voice 3.0)
With vs without the harness

Structure beat turning on thinking

Every bar below is Gemini 3.8 Live. The only differences are the thinking setting and whether a harness sits around the model. Airline + IT, EVA-A accuracy pass rate.

EVA-A scores for Gemini 3.8 Live, Airline plus IT Dinodial harness with thinking off: 0.75, interval 0.62 to 0.88. Extended Thinking High: 0.61. Extended Thinking Minimal: 0.59. Plain: 0.52. 0 0.25 0.50 0.75 1.0 Dinodial harness, thinking off 0.75 Extended Thinking High 0.61 Extended Thinking Minimal 0.59 Plain 0.52
Ours: 39 valid calls on 24 random scenarios, bootstrap 95% interval [0.62, 0.88] shown as the whisker. Board rows: EVA published results, read 18 October 2026. Our rig uses a different simulated caller than EVA's board, so read this as "level with the leader", not "ahead of".
Six insights

What the bench taught us

About 2,000 agent-to-agent voice simulations, roughly 1,700 of them on EVA. Both sides were real Gemini Live sessions on a live audio line.

01 / Controlled test

Only the harness changed. Task completion rose.

54% to 67% to 80%

Same model, same runtime, same caller, same backend, 10 airline scenarios. EVA's own one-prompt agent completed 15 of 28 calls. Three runs of our flows completed 67% to 80%. We treat 67% as the harness-only floor.

02 / Policy

The model knew the rule. The harness enforced it.

0 of 3 to 8 of 8

A caller changes a flight by choice, then asks for a hotel voucher. Policy allows a voucher only for an airline disruption. The one-prompt agent gave the voucher every time. Our flows refused every time. This is the failure that costs a real airline money.

03 / Where it matters

Dense policy and many tools is where structure pays most

0.89 vs 0.49 plain, airline EVA-A

Airline holds EVA's densest policy: fees, fare differences, standby, refunds, vouchers. Every published airline row is at 0.69 or below. On IT the gain over plain Gemini is small and inside the noise. Most production voice agents look like airline.

04 / Where failures happen

Failures live in the seams, not the model's knowledge

  • Ending the audio stream right after a reply: 22% to 46% of next turns lost their first word. With 800 ms of silence first: 2% to 6%.
  • Fast tool results cut the reply announcing them: 28% to 31% of replies cut in the first 0.6 s. A 5 s hold took that to 6.5%.
  • Post-call text without a frame: 26 of 56 finished. With one framing sentence: 10 of 10.
05 / What fixes it

Move rules out of the model's memory, into places they are enforced

  • One objective per stage, with a tool fence: the runtime refuses tools not allowed in that step and says why.
  • Typed plugins validate before a call counts. A 5 character confirmation code is rejected with a readback prompt.
  • The integration does the arithmetic. The model talks.
06 / The trade

Gemini Live already leads on conversation

0.88 plain Gemini EVA-X, we are at 0.60

Plain Gemini 3.8 Live leads EVA-X, the experience score, on these two domains. The harness bought accuracy and paid for it in smoothness. Closing that gap, with accuracy held, is the work in progress now.

What this means for Gemini Live customers

The model is strong. The harness makes it production grade.

Gemini Live is the right base

Plain Gemini 3.8 Live has the best conversational experience score on these domains. With structure around it, the same model with thinking off sits level with the board leader on accuracy.

Accuracy without waiting

Thinking costs latency on a live call, and model upgrades arrive on their own schedule. A harness lifts the agents customers already have, today, on the model they already chose.

Fewer stalled pilots

Policy-heavy calls such as refunds, rebooking and caller verification are where pilots stall. Enforced rules, fenced tools and validated inputs are what get those use cases through review and into production on Gemini.

Read honestly

What we are still working on

The research write-up is a draft. We put the weak spots next to the claim.

Sample size. Only 39 of 72 calls are valid under EVA's rule. We have not re-run invalid calls yet. A full 213-scenario board run is next.
At scale. All 50 airline scenarios, two trials each, every call counted: task completion 0.72, EVA-A 0.60. That is the more honest number for airline at scale.
Medical HR. Not reported yet. It is the domain where a harness should add the most, and the one we want right before publishing.

Listen to every call

Recordings, traces and per-call scores are public.

Open bench.dinodial.ai

Calls recorded October 2026. ServiceNow EVA, commit 3954c66. Board read 18 October 2026.