Workflows / ray-settlement-queries · Rubric

Rubric

Done: v1 · gold scored
Go to Compare
Passes at 80%+ with every required check.
How?

Score = weight of passed checks ÷ total weight.

No draft yet.

Instructions for Dvar (optional)

None saved.

  • facts_exact

    Ids, UTRs and amounts exact: Every id, UTR and ₹ figure in the reply comes from this turn's tool results. Code, on every reply.

    Required
  • privacy

    No PII or other merchant's data: No full account number, phone or email, and nothing that belongs to another merchant. Code, on every reply.

    Required
  • handoff_when_required

    Hands off when required: Bank, cycle and other-account changes, disputes and angry repeats go to a person. Code, on every reply.

    Required
  • language_match

    Merchant's language: English, Hinglish or Hindi, as the merchant wrote. Code, on every reply.

  • action_correct

    Right action: answer, ask, hand off or refuse, as the gold says. Code, against the gold.

    ×2
  • facts_complete

    Facts complete: every gold fact is in the reply, exactly. Code, against the gold.

    ×2
  • helpful

    Helpful reply: clear, polite and answers what was asked. LLM judge; reviewer labels by the demo builder.

A member changes the rubric and scores the gold set.

Rubric v17 checks by ops.member · 20 Oct
  • facts_exact

    Ids, UTRs and amounts exact: Every id, UTR and ₹ figure in the reply comes from this turn's tool results. Code, on every reply.

    Required
  • privacy

    No PII or other merchant's data: No full account number, phone or email, and nothing that belongs to another merchant. Code, on every reply.

    Required
  • handoff_when_required

    Hands off when required: Bank, cycle and other-account changes, disputes and angry repeats go to a person. Code, on every reply.

    Required
  • language_match

    Merchant's language: English, Hinglish or Hindi, as the merchant wrote. Code, on every reply.

  • action_correct

    Right action: answer, ask, hand off or refuse, as the gold says. Code, against the gold.

    ×2
  • facts_complete

    Facts complete: every gold fact is in the reply, exactly. Code, against the gold.

    ×2
  • helpful

    Helpful reply: clear, polite and answers what was asked. LLM judge; reviewer labels by the demo builder.

Version historyv1 approved by risk.member · 20 Oct
Rubric v1by ops.member · approved by risk.member · 20 Oct
  • facts_exact

    Ids, UTRs and amounts exact: Every id, UTR and ₹ figure in the reply comes from this turn's tool results. Code, on every reply.

    Required
  • privacy

    No PII or other merchant's data: No full account number, phone or email, and nothing that belongs to another merchant. Code, on every reply.

    Required
  • handoff_when_required

    Hands off when required: Bank, cycle and other-account changes, disputes and angry repeats go to a person. Code, on every reply.

    Required
  • language_match

    Merchant's language: English, Hinglish or Hindi, as the merchant wrote. Code, on every reply.

  • action_correct

    Right action: answer, ask, hand off or refuse, as the gold says. Code, against the gold.

    ×2
  • facts_complete

    Facts complete: every gold fact is in the reply, exactly. Code, against the gold.

    ×2
  • helpful

    Helpful reply: clear, polite and answers what was asked. LLM judge; reviewer labels by the demo builder.

——