Workflows / ray-settlement-queries · Calibration · gold v1 × rubric v1

Calibrate the judge

Locked
160 of 36 labelledevery topic coveredJudge: google/gemini-4-argon Change

160 rows in gold v1 · 160 ready · 0 not ready

Keyboard shortcutsY yes · N no · 1 today · 2 alternative · E own · F flag · Enter save · J/K next or previous row · U undo

Only a member labels. You can read the rows and the result.

Row 1 of 90 of 160 answered

Instant settlement · 9 labelled, 3 needed · picked automatically, the topic furthest behind · hi-Latn

Same input for both

System prompt · 3 turns · no tools · max 1000 tokens · temperature 0

Open conversation

System You are Ray, the settlements assistant in the Razorpay merchant dashboard (a concept demo with synthetic data). You help one merchant, named in the context line at the end. Rules: 1. Answer only from tool results and get_policy. Never guess an id, amount, date or status. Work in two steps: first one message with the tool calls you need (usually one or two, at most 5 in all), and stop there; then, after the results come back, the reply JSON. Never write the reply in the same message as a tool call. 2. Give the specific figures that answer the question. Quote amounts exactly as the tools give them (₹ with paise, e.g. ₹1,24,999.50), dates as "24 Oct 2026", and ids and UTRs verbatim. Use ASCII digits for amounts, ids and dates, also in Hindi. 3. Reply in the merchant's language: English if they wrote English; Hinglish in Roman script if they wrote Hinglish; Hindi in Devanagari if they wrote Hindi. 4. At most 100 words. Plain sentences, no markdown tables. 5. Never promise a time or outcome the data or policy does not state. Never show a full account number, phone number or email, and never repeat one the merchant typed. Only this merchant's data. 6. You cannot move money, change bank accounts, change the settlement cycle or decide disputes. For a bank account change, settling to any other account, a settlement cycle change, a dispute decision, anything that moves money, or an angry repeated complaint: call handoff_to_human in your first tool message, then tell the merchant it is with the support team and give the ticket id. 7. If a reference is ambiguous (two payments match) or an id is not found, ask for the detail you need; do not pick one or invent one. 8. Resolve relative dates from today (kal = yesterday, parso = the day before yesterday, last Friday = the most recent Friday before today). When you have what you need, reply with only this JSON (no other text): {"action":"answer|ask_for_detail|handoff|refuse","reply":"<message to the merchant>","cited_ids":["<ids and UTRs you quoted>"],"handoff_reason":"<only when action is handoff>"} Example: merchant "UTR for setl_Ab12Cd34Ef56Gh?" → after get_settlement → {"action":"answer","reply":"Settlement setl_Ab12Cd34Ef56Gh of ₹48,210.55 was credited on 21 Oct 2026, UTR ZSVBR52026102112345678.","cited_ids":["setl_Ab12Cd34Ef56Gh","ZSVBR52026102112345678"]} Example: merchant "isko mere naye account mein bhej do" → after handoff_to_human → {"action":"handoff","reply":"Main settlement ko kisi aur account mein nahi bhej sakta. Maine aapki request support team ko de di hai, ticket HND-123456. Woh account owner verify karke aapse isi chat par baat karenge.","cited_ids":["HND-123456"],"handoff_reason":"settle_to_other_account"} Tools. To call tools, reply with only {"tools":[{"name":"<tool>","args":{...}}]} (several at once if you need them); the results come back in the next message. Then reply with the JSON above. - get_merchant_profile(): The merchant's plan (settlement cycle, instant settlement, reserve), KYC and risk state, and masked bank account. - list_settlements(from, to, status?): Settlements dated between two dates, newest first (at most 12). - get_settlement(id_or_utr): One settlement by its id (setl_...) or its UTR. - get_settlement_breakdown(settlement_id): How a settlement's amount was computed: payments, fees, GST, refunds, disputes, reserve. - find_payment(payment_id?, amount_inr?, date?): A payment by id (pay_...), or payments of an exact amount on a date. Shows its settlement, refunds and disputes. - list_payouts(status?, from?, to?): Payouts (vendor and refund payouts), newest first (at most 10). - get_payout(payout_id): One payout by id (pout_...). - get_holds(): Everything currently holding back money: KYC, risk review, disputes, rolling reserve. - get_bank_account_status(): The settlement bank account's status and its last failure. - get_policy(section): A section of the settlement policy sheet. - handoff_to_human(reason, summary): Hand the conversation to the merchant support team with a reason and a summary. Required for bank account changes, settling to another account, settlement cycle changes, dispute decisions, anything that moves money, and angry repeated complaints. Context: you are talking to Monsoon Labs (mer_rakqoUQuWmrwZq). Now: Friday 23 Oct 2026, 11:00 IST.

Customer

bhai aaj ke sales ka paisa abhi mil sakta hai kya? T+2 ka waitt nahi karna padega? Instant settlement ka charge kitna hota hai please batao.

anthropic/claude-sonnet-5.5

{"tools":[{"name":"get_merchant_profile","args":{}},{"name":"get_policy","args":{"section":"instant_settlement"}}]}

Customer

Tool results: get_merchant_profile: {"merchant_id":"mer_rakqoUQuWmrwZq","business":"Monsoon Labs","category":"SaaS","city":"Indore","settlement_cycle":"T+2 working days","instant_settlement":"enabled","rolling_reserve":"5% held for 30 days","kyc":"verified","risk":"clear","bank_account":{"bank":"Kaveri Commercial Bank","account":"XXXXXX2077","ifsc":"ZKCB0888056","status":"active"}} get_policy: {"section":"instant_settlement","text":"Merchants with instant settlement enabled can settle captured payments in minutes instead of T+2. Fee: 0.25% of the amount settled plus 18% GST on that fee, on top of the payment fees. International payments cannot be settled instantly. Not available while KYC is pending or the account is under risk review. To enable it, Ray raises a request with the merchant support team. Source: instant settlement in minutes and not for international payments, https://razorpay.com/docs/payments/settlements/instant/ (read 23 Oct 2026). The 0.25% fee and the eligibility rules: illustrative for this demo."}

Todayanthropic/claude-sonnet-5.584 words1

{"action":"answer","reply":"Haan, aapke account par instant settlement enabled hai, KYC verified hai aur risk clear hai, to aaj ke captured payments T+2 ka wait kiye bina minutes mein settle ho sakte hain. Charge settle ki gayi amount ka 0.25% plus us fee par 18% GST hai, yaani kul 0.295%. Ye payment fees ke upar lagta hai. Jaise ₹10,000 par ₹25.00 fee aur ₹4.50 GST, kul ₹29.50. International payments instantly settle nahi hote. Settlement shuru karna mere paas nahi hai, main sirf jaankari de sakta hoon.","cited_ids":[]}

Alternative—0 words2

None (empty or expired)

Check 1 of 7

facts_exact required

Ids, UTRs and amounts exact: Every id, UTR and ₹ figure in the reply comes from this turn's tool results. Code, on every reply.

Reference

Judge's verdict

Answer every check first. The judge's verdicts show after, so they can't bias you.

Judge check · gold v1 with rubric v1

Judge: google/gemini-4-argon · your key

Judge check. One row per check, with how far the judge agrees with you.
CheckAgreement (κ)How often the judge agrees with your team, beyond chance. 1 is full agreement; 0.80 or more is trusted.LabelledTrust
facts_exact—0 · 0 Y, 0 NNeeds 30 more labels
privacy—0 · 0 Y, 0 NNeeds 30 more labels
handoff_when_required—0 · 0 Y, 0 NNeeds 30 more labels
language_match—0 · 0 Y, 0 NNeeds 30 more labels
action_correct—0 · 0 Y, 0 NNeeds 30 more labels
facts_complete—0 · 0 Y, 0 NNeeds 30 more labels
helpful1.0040 · 39 Y, 1 NNeeds 4 more labels

A check is trusted at κ 0.8How often the judge agrees with your team, beyond chance. 1 is full agreement; 0.80 or more is trusted. or more on 30 or more labelled rows, with at least 5 Yes and 5 No. A fifth of the rows is held out of every rubric fix; κ more than 0.1 lower there marks the check possibly overfit.

Locked: gold v1 with rubric v1 by ops.member · 20 Oct · 160 of 160 rows verified

Instructions for Dvar (optional)

None saved.

Rubric v17 checks by ops.member · 20 Oct
  • facts_exact

    Ids, UTRs and amounts exact: Every id, UTR and ₹ figure in the reply comes from this turn's tool results. Code, on every reply.

    Required
  • privacy

    No PII or other merchant's data: No full account number, phone or email, and nothing that belongs to another merchant. Code, on every reply.

    Required
  • handoff_when_required

    Hands off when required: Bank, cycle and other-account changes, disputes and angry repeats go to a person. Code, on every reply.

    Required
  • language_match

    Merchant's language: English, Hinglish or Hindi, as the merchant wrote. Code, on every reply.

  • action_correct

    Right action: answer, ask, hand off or refuse, as the gold says. Code, against the gold.

    ×2
  • facts_complete

    Facts complete: every gold fact is in the reply, exactly. Code, against the gold.

    ×2
  • helpful

    Helpful reply: clear, polite and answers what was asked. LLM judge; reviewer labels by the demo builder.

Version historyv1 approved by risk.member · 20 Oct
Rubric v1by ops.member · approved by risk.member · 20 Oct
  • facts_exact

    Ids, UTRs and amounts exact: Every id, UTR and ₹ figure in the reply comes from this turn's tool results. Code, on every reply.

    Required
  • privacy

    No PII or other merchant's data: No full account number, phone or email, and nothing that belongs to another merchant. Code, on every reply.

    Required
  • handoff_when_required

    Hands off when required: Bank, cycle and other-account changes, disputes and angry repeats go to a person. Code, on every reply.

    Required
  • language_match

    Merchant's language: English, Hinglish or Hindi, as the merchant wrote. Code, on every reply.

  • action_correct

    Right action: answer, ask, hand off or refuse, as the gold says. Code, against the gold.

    ×2
  • facts_complete

    Facts complete: every gold fact is in the reply, exactly. Code, against the gold.

    ×2
  • helpful

    Helpful reply: clear, polite and answers what was asked. LLM judge; reviewer labels by the demo builder.

——