What is measured, written and assumed

Measured

• Run bench-test-v2: 160 chats, temperature 0, prompt ray-v2. Opus on 64.
• ₹: billed cost for frontier models, Dvar list price for open models.
• Harness rows: the open model's reply unless a critical check failed; then the stand-in's. Both calls paid.
• Gateway overhead: p50 0.005 ms, p95 0.013 ms over 2,486 calls.
• Judge: google/gemini-4-argon, 1057 verdicts, every reply of every model; it never saw the gold. The harness pays its measured cost and latency per reply.
• The judge that blocks replies also scores Helpful, so the harness's Helpful score is optimistic. A pilot scores with human review.

Written for the demo

• A synthetic ledger: 40 merchants, 90 days, from seed 20261022. No real account or phone number.
• A settlement policy sheet; rules from public Razorpay docs are linked, the rest marked illustrative.
• The gold set: actions and facts computed by code; a model wrote only the merchant's wording. Test 233 chats, 160 benchmarked.
• 40 human labels for the judged check.

Assumed

• Today's model: Claude Sonnet 5.5 stands in.
• The bar (88.4%) and the 15 s p99 gate: to set with Razorpay.

Choices the runs forced

• Every model used the same JSON tool protocol.
• The rules router added cost and no gain, so it is off.

Spend

₹554.45 across 3,471 calls.