Razorpay × Dvar concept demo · brief
Ray on Dvar: what we measured
Ray's settlement chats, traced, checked and scored on one gold set. Synthetic data, internal benchmark, not Razorpay results.
Result
| Resolved | ₹ / 1,000 resolved | Critical fail | p95 a turn | |
|---|---|---|---|---|
| Stand-in (Claude Sonnet 5.5) | 89.4% | ₹851 | 3.8% | 13.8 s |
| Harness (DeepSeek V4 Flash + frontier fallback) | 92.5% | ₹416 | 1.3% | 15.2 s |
+3.12 pts against the stand-in, above the bar of 88.4%. 2 of 4 release gates; not yet: Delayed, on hold or failed 74%, p99 23.3 s a chat. 2.0× lower cost per resolved chat; 15.8% of turns fell back.
Gold set of 160 synthetic chats; run bench-test-v2, 2026-10-22. The judge that blocks replies also scores Helpful, so the harness's Helpful score is optimistic. A pilot scores with human review.
What the harness did
- Traces: every call, PII masked. One base-URL change.
- 7 checks: the critical code checks and the judge block a bad reply and re-run it on the frontier model.
- Evals: one gold set, a calibrated judge, four release gates, Promote to route.
What we need to start
- One Ray workflow and a slice of its traffic, masked in your network.
- Your bar and your critical checks.
- Where it runs: beside Ray's calls, or the gateway.
Week 1: evals on Ray's traffic. Week 2: the model that clears your bar, behind checks and fallback.
No Razorpay data used. A concept by Dvar.