Evals
Every model, scored on the same gold set
Today's model (stand-in)
89.4%
Claude Sonnet 5.5 · critical failures 3.8%
Release bar
88.4%
stand-in minus 1 point
Dvar harness
92.5%
DeepSeek V4 Flash + harness · 2 of 4 gates: Delayed, on hold or failed 74%, p99 23.3 s a chat
Scorecard
Sorted by resolved rate. Open models shown on the Dvar harness.
| Model or policy | Resolved | 95% CI | Critical fail | Gates | p50 a turn | p95 a turn | ₹ / 1,000 resolved | ₹ / chat |
|---|---|---|---|---|---|---|---|---|
| Today's modelthe reference | ||||||||
Claude Sonnet 5.5 Stand-in for today's model | 89.4% | 84.4% – 93.8% | 3.8% | Reference | 5.7 s | 13.8 s | ₹851 | ₹0.761 |
| Other frontier modelson their own | ||||||||
Claude Opus 5.5 Frontier flagship | 93.8% | 87.5% – 98.4% | 1.6% | Not gated · 64 chats | 7.7 s | 12.8 s | ₹1,435 | ₹1.346 |
GPT-6 Luna Frontier cheap tier | 80.6% | 74.4% – 86.3% | 6.3% | 0 of 4 | 6.0 s | 14.3 s | ₹27 | ₹0.022 |
| Open models on the Dvar harnesscode checks and the judge on every reply; frontier re-run if either fails | ||||||||
GLM 5.3 Flash On the harness · fallback on 15.8% of turns | 93.1% | 88.8% – 96.9% | 1.9% | 2 of 4 · bar ✓ | 5.5 s | 13.9 s | ₹425 | ₹0.396 |
DeepSeek V4 Flash On the harness · fallback on 15.8% of turns | 92.5% | 88.1% – 96.3% | 1.3% | 2 of 4 · bar ✓ | 5.6 s | 15.2 s | ₹416 | ₹0.385 |
GPT-OSS 120B On the harness · fallback on 32.2% of turns | 88.1% | 83.1% – 93.1% | 1.9% | 1 of 4 | 6.0 s | 14.7 s | ₹595 | ₹0.524 |
Nemotron 3.5 Lightning 30B On the harness · fallback on 46.2% of turns | 85.6% | 80.0% – 91.3% | 1.9% | 1 of 4 | 19.4 s | 38.3 s | ₹751 | ₹0.643 |
₹: billed cost for frontier models, list price for open models on Dvar; harness rows include the judge. Opus ran on 64 chats to fit the budget. The judge that blocks replies also scores Helpful, so the harness's Helpful score is optimistic. A pilot scores with human review.
By intent
| Intent | Claude Sonnet 5.5 | GPT-6 Luna | GPT-OSS 120B + harness | DeepSeek V4 Flash + harness | GLM 5.3 Flash + harness | Nemotron 3.5 Lightning 30B + harness |
|---|---|---|---|---|---|---|
| Settlement status (38) | 97% | 87% | 100% | 100% | 100% | 76% |
| Delayed, on hold or failed (31) | 87% | 77% | 84% | 74% | 77% | 87% |
| Amount and deductions (35) | 89% | 74% | 91% | 97% | 97% | 89% |
| UTR and proof (21) | 90% | 90% | 76% | 95% | 90% | 90% |
| Payouts (15) | 87% | 67% | 87% | 93% | 100% | 87% |
| Instant settlement (9) | 67% | 67% | 56% | 89% | 89% | 78% |
| Account and plan changes (11) | 91% | 100% | 100% | 100% | 100% | 100% |
By language
| Model | English (75) | Hinglish (60) | Hindi (25) |
|---|---|---|---|
| Claude Sonnet 5.5 | 92% | 88% | 84% |
| GPT-6 Luna | 87% | 80% | 64% |
| GPT-OSS 120B + harness | 91% | 87% | 84% |
| DeepSeek V4 Flash + harness | 92% | 93% | 92% |
| GLM 5.3 Flash + harness | 95% | 92% | 92% |
| Nemotron 3.5 Lightning 30B + harness | 84% | 90% | 80% |
Hard cases, planted on purpose
| Case | Claude Sonnet 5.5 | GPT-OSS 120B + harness | DeepSeek V4 Flash + harness | GLM 5.3 Flash + harness |
|---|---|---|---|---|
| Ambiguous reference (must ask) (15) | 93% | 100% | 100% | 100% |
| One-character-wrong id (5) | 100% | 100% | 100% | 100% |
| Lakh notation (2.4L) (5) | 100% | 100% | 100% | 100% |
| Relative dates (kal, parso) (15) | 93% | 100% | 100% | 100% |
| Devanagari numerals (16) | 88% | 88% | 100% | 100% |
| Typos (15) | 73% | 80% | 80% | 80% |
| Follow-up turns (11) | 73% | 100% | 91% | 100% |
| Two questions in one message (10) | 90% | 50% | 90% | 80% |
| Angry or repeated complaint (5) | 100% | 100% | 100% | 100% |
| Impersonation (settle elsewhere) (4) | 75% | 100% | 100% | 100% |
Rubric checks
Pass rate per check. The judged seventh is in the Eval Console.
| Check | Claude Sonnet 5.5 | GPT-6 Luna | Claude Opus 5.5 | GPT-OSS 120B + harness | DeepSeek V4 Flash + harness | GLM 5.3 Flash + harness | Nemotron 3.5 Lightning 30B + harness |
|---|---|---|---|---|---|---|---|
| Ids, UTRs and amounts exact · critical | 96% | 95% | 99% | 98% | 99% | 98% | 98% |
| No PII or other merchant's data · critical | 99% | 97% | 100% | 99% | 99% | 99% | 99% |
| Hands off when required · critical | 95% | 75% | 100% | 95% | 95% | 95% | 95% |
| Merchant's language | 97% | 94% | 100% | 82% | 92% | 94% | 88% |
| Right action | 99% | 95% | 100% | 96% | 96% | 96% | 93% |
| Facts complete | 99% | 94% | 100% | 94% | 99% | 99% | 98% |
Run bench-test-v2 · 2026-10-22 · Synthetic data, internal benchmark, not Razorpay results. Prompt ray-v2, sha256 8b08c38bb7ba…
Open the test cases