{
  "created": "2026-11-19",
  "note": "Answer key written before any model call. Rules are in the system prompt.",
  "system_triage": "You triage customer support messages. For each message return one JSON object with exactly these keys:\n- \"id\": the message id as given.\n- \"category\": one of \"billing\", \"bug\", \"account_access\", \"feature_request\", \"shipping\".\n- \"order_id\": the first valid order ID in the message, normalised to uppercase, or null. A valid order ID is \"ORD-\" followed by exactly 6 digits (case-insensitive). Ticket numbers, phone numbers and IDs with a different digit count are not order IDs.\n- \"refund\": true only if the customer explicitly asks for money back (refund, money back, chargeback threat). Otherwise false, including when they say they do not want a refund.\nCategory rules: non-delivery, late delivery, tracking and items damaged in transit are \"shipping\" even if a refund is requested. Charges, invoices, plans and refunds not caused by delivery are \"billing\". Login problems caused by passwords, 2FA, email changes or a locked account are \"account_access\"; app malfunctions and error pages are \"bug\", even when they appear on a billing page. Requests for new functionality are \"feature_request\". If more than one category applies, use the category of the issue mentioned first.\nReturn only a JSON array of the objects, in the input order, with no other text.",
  "items": [
    {"id": 1, "text": "I was charged twice for ORD-482913 this morning. Please refund the duplicate charge.", "gold": {"category": "billing", "order_id": "ORD-482913", "refund": true}},
    {"id": 2, "text": "The export button crashes the app on Android 15 every time. No order involved.", "gold": {"category": "bug", "order_id": null, "refund": false}},
    {"id": 3, "text": "Can't log in. The password reset email never arrives.", "gold": {"category": "account_access", "order_id": null, "refund": false}},
    {"id": 4, "text": "Would love a dark mode for the dashboard.", "gold": {"category": "feature_request", "order_id": null, "refund": false}},
    {"id": 5, "text": "Package for ord-551204 still says 'label created' after 9 days. Where is it?", "gold": {"category": "shipping", "order_id": "ORD-551204", "refund": false}},
    {"id": 6, "text": "Not asking for a refund, I just want the invoice for ORD-300017 to show my company VAT number.", "gold": {"category": "billing", "order_id": "ORD-300017", "refund": false}},
    {"id": 7, "text": "Order ORD-77812 never arrived. I want my money back.", "gold": {"category": "shipping", "order_id": null, "refund": true}},
    {"id": 8, "text": "Your app logs me out every 5 minutes since the last update. Ticket TCK-204411.", "gold": {"category": "bug", "order_id": null, "refund": false}},
    {"id": 9, "text": "Call me at 415-555-0199 about ORD-918273. The lamp arrived smashed and I'd like a refund.", "gold": {"category": "shipping", "order_id": "ORD-918273", "refund": true}},
    {"id": 10, "text": "Pedido ORD-640022: me cobraron el plan anual pero pedí el mensual. Quiero el reembolso de la diferencia.", "gold": {"category": "billing", "order_id": "ORD-640022", "refund": true}},
    {"id": 11, "text": "Great, another 'upgrade' that broke CSV import. Fantastic work.", "gold": {"category": "bug", "order_id": null, "refund": false}},
    {"id": 12, "text": "Please add an API endpoint to bulk-delete projects. Also, small thing: ORD-100200 is fine, no issues there.", "gold": {"category": "feature_request", "order_id": "ORD-100200", "refund": false}},
    {"id": 13, "text": "My 2FA codes stopped working after I changed phones and now I'm locked out.", "gold": {"category": "account_access", "order_id": null, "refund": false}},
    {"id": 14, "text": "I'm cancelling. Refund my last month (ORD-203040) or I'll file a chargeback.", "gold": {"category": "billing", "order_id": "ORD-203040", "refund": true}},
    {"id": 15, "text": "Tracking for ORD-55120 and ORD-551209 both show delivered, but only one box came.", "gold": {"category": "shipping", "order_id": "ORD-551209", "refund": false}},
    {"id": 16, "text": "It would be helpful if invoices could be emailed to two addresses.", "gold": {"category": "feature_request", "order_id": null, "refund": false}},
    {"id": 17, "text": "The page shows 'Error 502' whenever I open billing settings.", "gold": {"category": "bug", "order_id": null, "refund": false}},
    {"id": 18, "text": "I haven't been charged for ORD-712345 yet. Is that normal? Don't refund anything, I want to keep the order.", "gold": {"category": "billing", "order_id": "ORD-712345", "refund": false}},
    {"id": 19, "text": "Order #ORD-404404 hasn't shipped and it's been two weeks.", "gold": {"category": "shipping", "order_id": "ORD-404404", "refund": false}},
    {"id": 20, "text": "Someone changed my account email without permission and I can't sign in. Also refund ORD-909090, I didn't place it.", "gold": {"category": "account_access", "order_id": "ORD-909090", "refund": true}}
  ],
  "reasoning": [
    {"id": "R1", "prompt": "A batch job sends 12,000 requests. Each request uses 1,800 input tokens and 250 output tokens. Input costs $0.10 per million tokens and output costs $0.50 per million tokens. 4% of the requests fail validation and are retried exactly once with the same token usage; retries always succeed. What is the total cost in USD? Show brief working, then put the final number rounded to 4 decimal places on the last line as: ANSWER: <number>", "gold": "3.8064", "check": "number"},
    {"id": "R2", "prompt": "Business hours are Monday to Friday, 09:00 to 17:00 local time; no other time counts toward an SLA. A ticket is created on Friday at 15:00 with a 6 business-hour SLA. A second ticket is created on Monday at 22:30 with an 8 business-hour SLA. Give each deadline as weekday and HH:MM. Put the final answer on the last line as: ANSWER: <ticket1 deadline>; <ticket2 deadline>", "gold": "Monday 13:00; Tuesday 17:00", "check": "sla"},
    {"id": "R3", "prompt": "Refund policy: (a) Physical items that are unused can be refunded within 30 days of delivery. (b) Items damaged on arrival can be refunded within 90 days of delivery. (c) Digital goods can be refunded at any time if never downloaded; otherwise they are not refundable. All requests below were made on November 15, 2026. Request 1: an e-book bought October 1, 2026, never downloaded. Request 2: an unused jacket delivered October 10, 2026. Request 3: a lamp delivered September 1, 2026 that arrived damaged. For each request, decide eligible or not eligible. Put the final answer on the last line as: ANSWER: <1>, <2>, <3> using yes or no.", "gold": "yes, no, yes", "check": "yesno"}
  ]
}
