RazorpayDvar

Razorpay × Dvar concept demo

See every Ray reply, check it in code, and change models only when the evals say so.

Ray answering merchants' settlement questions, run through Dvar's observability and eval harness. English, Hinglish and Hindi.

  • Every call traced: each step, tool call, ₹ and latency, PII masked. One line to connect.
  • 7 checks on every reply: the code checks and the judge block a bad reply before the merchant sees it.
  • Evals gate every model change: 7 models on one gold set; only a model that clears the gates ships.
  • Open models, measured: DeepSeek V4 Flash on the harness resolves 92.5% against 89.4% today, at 2.0× lower cost. 2 of 4 release gates.
Start the guided tourTalk to RaySee the scorecard

The loop

Calls become traces, traces become evals, evals set the route.

  1. 1

    Traffic

    Ray's chats

  2. 2

    Gateway

    Routes each step

  3. 3

    Traces

    Every call, PII masked

  4. 4

    Evals

    Gold set, 7 checks, gates

  5. 5

    Route

    Promote what passes

Glossary

Resolved chat
Right action, every gold fact exact, no critical failure, every reply judged helpful.
Critical check
Blocks the reply on failure: facts exact, privacy, hand-off.
Fallback
A reply blocked by a critical check or the judge re-runs once on the frontier model. The draft becomes an eval case.
Stand-in
Today's model, stood in by Claude Sonnet 5.5: we don't know Ray's.
Gold set
The expected action and exact facts per chat. Here computed from a synthetic ledger; in production, sampled from Ray's traffic.
The bar
The stand-in's resolved rate minus 1 point. Set with Razorpay in a pilot.
Gateway
One OpenAI-compatible endpoint per workflow. Routes each step, traces every call, never fails a call over tracing.
Promote
Writes a model that passed the gates into the gateway's route.

Built for ₹554.45 of model calls. Synthetic data, internal benchmark, not Razorpay results.