| Model | Quality | p99 latency | Cost per request | vs today | Result |
|---|---|---|---|---|---|
| Claude Sonnet 5.5 Stand-in for today's modelopenrouter/anthropic/claude-sonnet-5.5 | 89.4% | 21,191 ms | ₹0.7609 | — | Today |
| openrouter/openai/gpt-6-luna | 80.6% | 21,553 ms | ₹0.0221 | −97% | Fails 4 checks |
openai/gpt-6-luna is fully right on 80.6% of gold rows, under the 88.4% quality bar; has a critical failure (an invented id or amount, a PII leak, another merchant's data, or a missed handoff) on 10 gold rows where today's model has none; falls below the minimum on Settlement status: 0.87 vs 0.88 (38 gold rows); Delayed, on hold or failed: 0.77 vs 0.88 (31 gold rows); Amount and deductions: 0.74 vs 0.88 (35 gold rows); Payouts: 0.67 vs 0.88 (15 gold rows). Every other gated topic is at or above its minimum; is too slow: p99 21,553 ms replayed on 160 gold rows, over the 15,000 ms limit. Weakest topic: Payouts 0.67 | |||||
| openai_compatible/dvar:deepseek-v4-flash-0731 | 78.8% | 8,853 ms | ₹0.0439 | −94% | Fails 3 checks |
dvar:deepseek-v4-flash-0731 is fully right on 78.8% of gold rows, under the 88.4% quality bar; has a critical failure (an invented id or amount, a PII leak, another merchant's data, or a missed handoff) on 9 gold rows where today's model has none; falls below the minimum on Delayed, on hold or failed: 0.68 vs 0.88 (31 gold rows); Amount and deductions: 0.83 vs 0.88 (35 gold rows); UTR and proof: 0.76 vs 0.88 (21 gold rows); Payouts: 0.53 vs 0.88 (15 gold rows); Account and plan changes: 0.82 vs 0.88 (11 gold rows). Every other gated topic is at or above its minimum. Weakest topic: Payouts 0.53 | |||||
| openai_compatible/dvar:glm-5.3-flash | 79.4% | 8,828 ms | ₹0.0544 | −93% | Fails 3 checks |
dvar:glm-5.3-flash is fully right on 79.4% of gold rows, under the 88.4% quality bar; has a critical failure (an invented id or amount, a PII leak, another merchant's data, or a missed handoff) on 8 gold rows where today's model has none; falls below the minimum on Settlement status: 0.84 vs 0.88 (38 gold rows); Delayed, on hold or failed: 0.71 vs 0.88 (31 gold rows); Amount and deductions: 0.86 vs 0.88 (35 gold rows); UTR and proof: 0.71 vs 0.88 (21 gold rows); Payouts: 0.60 vs 0.88 (15 gold rows). Every other gated topic is at or above its minimum. Weakest topic: Payouts 0.60 | |||||
| openai_compatible/dvar:gpt-oss-120b | 58.8% | 5,524 ms | ₹0.0563 | −93% | Fails 3 checks |
dvar:gpt-oss-120b is fully right on 58.8% of gold rows, under the 88.4% quality bar; has a critical failure (an invented id or amount, a PII leak, another merchant's data, or a missed handoff) on 4 gold rows where today's model has none; falls below the minimum on Settlement status: 0.68 vs 0.88 (38 gold rows); Delayed, on hold or failed: 0.61 vs 0.88 (31 gold rows); Amount and deductions: 0.57 vs 0.88 (35 gold rows); UTR and proof: 0.57 vs 0.88 (21 gold rows); Payouts: 0.33 vs 0.88 (15 gold rows); Account and plan changes: 0.73 vs 0.88 (11 gold rows). Weakest topic: Payouts 0.33 | |||||
| openai_compatible/dvar:nemotron-3.5-lightning-30b-a3b-bf16 | 43.1% | 40,153 ms | ₹0.0711 | −91% | Fails 4 checks |
dvar:nemotron-3.5-lightning-30b-a3b-bf16 is fully right on 43.1% of gold rows, under the 88.4% quality bar; has a critical failure (an invented id or amount, a PII leak, another merchant's data, or a missed handoff) on 28 gold rows where today's model has none; falls below the minimum on Settlement status: 0.24 vs 0.88 (38 gold rows); Delayed, on hold or failed: 0.45 vs 0.88 (31 gold rows); Amount and deductions: 0.51 vs 0.88 (35 gold rows); UTR and proof: 0.76 vs 0.88 (21 gold rows); Payouts: 0.27 vs 0.88 (15 gold rows); Account and plan changes: 0.45 vs 0.88 (11 gold rows); is too slow: p99 40,153 ms replayed on 160 gold rows, over the 15,000 ms limit. Weakest topic: Settlement status 0.24 | |||||
| openai_compatible/dvar-harness:deepseek-v4-flash-0731 | 92.5% | 23,295 ms | ₹0.3846 | −49% | Fails 2 checks |
dvar-harness:deepseek-v4-flash-0731 falls below the minimum on Delayed, on hold or failed: 0.74 vs 0.88 (31 gold rows). Every other gated topic is at or above its minimum; is too slow: p99 23,295 ms replayed on 160 gold rows, over the 15,000 ms limit. Weakest topic: Delayed, on hold or failed 0.74 | |||||
Post-trained Ray model: next step, measured after training on Ray's outcomes. No numbers yet. It goes through this same gate: Fine-tune. | |||||
No candidate passes every check. Open a model's row for its reasons.
Prices are per request, at each provider's list price.
Judge: 0 of 7 checks trusted.
| Topic | Rows | Today · Claude Sonnet 5.5 | Candidate · GPT-6 Luna | Candidate · DeepSeek V4 Flash | Candidate · GLM 5.3 Flash | Candidate · GPT-OSS 120B | Candidate · Nemotron 3.5 Lightning 30B | Candidate · DeepSeek V4 Flash + Dvar harness | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Settlement status | 38 | 0.97 | 0.87 | −0.10 | 0.92 | −0.05 | 0.84 | −0.13 | 0.68 | −0.29 | 0.24 | −0.73 | 1.00 | +0.03 |
| Amount and deductions | 35 | 0.89 | 0.74 | −0.15 | 0.83 | −0.06 | 0.86 | −0.03 | 0.57 | −0.32 | 0.51 | −0.38 | 0.97 | +0.08 |
| Delayed, on hold or failed | 31 | 0.87 | 0.77 | −0.10 | 0.68 | −0.19 | 0.71 | −0.16 | 0.61 | −0.26 | 0.45 | −0.42 | 0.74 | −0.13 |
| UTR and proof | 21 | 0.90 | 0.90 | 0.00 | 0.76 | −0.14 | 0.71 | −0.19 | 0.57 | −0.33 | 0.76 | −0.14 | 0.95 | +0.05 |
| Payouts | 15 | 0.87 | 0.67 | −0.20 | 0.53 | −0.34 | 0.60 | −0.27 | 0.33 | −0.54 | 0.27 | −0.60 | 0.93 | +0.06 |
| Account and plan changes | 11 | 0.91 | 1.00 | +0.09 | 0.82 | −0.09 | 1.00 | +0.09 | 0.73 | −0.18 | 0.45 | −0.46 | 1.00 | +0.09 |
| Instant settlement | 9 | 0.67 | 0.67 | 0.00 | 0.89 | +0.22 | 0.89 | +0.22 | 0.44 | −0.23 | 0.33 | −0.34 | 0.89 | +0.22 |
| All topics, weighted by traffic | 160 | 0.89 | 0.81 | −0.08 | 0.79 | −0.10 | 0.79 | −0.10 | 0.59 | −0.30 | 0.43 | −0.46 | 0.92 | +0.03 |