Measured Index #1: run index-1-2026-11-20 ========================================================================== started (UTC) : 2026-11-20T15:11:28+00:00 harness : scripts/index_run.py anthropic : 1.2.0 pydantic: 2.13.5 python : 3.12.16 on Linux-7.0.0-1019-azure-x86_64-with-glibc2.43 book default : claude-opus-5 (BOOK.md, Course Module 1 harness row) prices : https://platform.claude.com/docs/en/about-claude/pricing (read 2026-11-20), USD per million tokens limits/model : 80 calls, 120000 tokens; max_tokens single/chain 1024, typed 1000, agent 2048; agent steps 5 suites : route-single (10), route-chain (10), route-typed (10), agent (5) models the key lists (15): claude-fable-5, claude-fable-5-1, claude-fable-5-2, claude-haiku-4-5-20251001, claude-haiku-5-5, claude-opus-4-5-20251101, claude-opus-4-6, claude-opus-4-7, claude-opus-4-8, claude-opus-5, claude-opus-5-5, claude-sonnet-4-5-20250929, claude-sonnet-4-6, claude-sonnet-5, claude-sonnet-5-5 claude-fable-5-2 callable claude-opus-5-5 callable claude-sonnet-5-5 callable claude-haiku-5-5 callable claude-opus-5 callable claude-opus-4-6 callable claude-haiku-4-5-20251001 callable listed, not in Issue 1's plan: claude-fable-5, claude-fable-5-1, claude-opus-4-5-20251101, claude-opus-4-7, claude-opus-4-8, claude-sonnet-4-5-20250929, claude-sonnet-4-6, claude-sonnet-5 -------------------------------------------------------------------------- model claude-fable-5-2 -------------------------------------------------------------------------- route-single (1 call per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. OK got 'general' route-chain (3 calls per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. MISS got 'auth' route-typed (1 call per ticket, messages.parse) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. MISS got 'auth' agent (up to 5 steps per question) 1. OK got 'There are **7 headsets** in stock (SKU HS-03), priced at $89' 2. OK got '12 × 45 = **540**' 3. OK got 'The total value of webcams on hand is **$186.00** (3 units o' 4. OK got 'You have more keyboards: **12 keyboards** (KB-01) on hand ve' 5. OK got 'Bringing mice (SKU MS-02) up to 10 units would cost **$195.0' 6. -- got 'For slow movers, the usual goal is to minimize cash tied up ' -------------------------------------------------------------------------- model claude-opus-5-5 -------------------------------------------------------------------------- route-single (1 call per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. OK got 'general' route-chain (3 calls per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. MISS got 'auth' route-typed (1 call per ticket, messages.parse) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. MISS got 'auth' agent (up to 5 steps per question) 1. OK got 'There are 7 headsets in stock (SKU HS-03). They cost $89.00 ' 2. OK got '12 × 45 = **540**' 3. OK got 'The webcams on hand are worth **$186.00** in total. There ar' 4. OK got 'We have more keyboards. There are **12 keyboards** (SKU KB-0' 5. OK got 'It would cost **$195.00** to bring mice up to 10 units. - *' 6. -- got 'For slow-moving items, the main goal is to keep the right pa' -------------------------------------------------------------------------- model claude-sonnet-5-5 -------------------------------------------------------------------------- route-single (1 call per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. OK got 'general' route-chain (3 calls per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. MISS got 'auth' route-typed (1 call per ticket, messages.parse) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. MISS got 'auth' agent (up to 5 steps per question) 1. OK got "There are 7 headsets in stock (SKU HS-03). They're priced at" 2. OK got '12 × 45 = 540.' 3. OK got 'The webcams on hand (SKU WC-04) are worth a total of **$186.' 4. OK got 'You have more keyboards than headsets. There are 12 keyboard' 5. OK got 'Bringing mice (SKU MS-02) up to 10 units would cost **$195.0' 6. -- got '# Reorder Policy for Slow-Moving Items The goal is to avoid' -------------------------------------------------------------------------- model claude-haiku-5-5 -------------------------------------------------------------------------- route-single (1 call per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. OK got 'general' route-chain (3 calls per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. MISS got 'auth' route-typed (1 call per ticket, messages.parse) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. MISS got 'auth' agent (up to 5 steps per question) 1. OK got 'There are 7 headsets in stock (SKU HS-03).' 2. OK got '12 × 45 = 540.' 3. OK got 'The total value of webcams on hand is **$186.00**. The stock' 4. OK got 'We have more keyboards than headsets: 12 keyboards (SKU KB-0' 5. OK got 'Bringing mice up to 10 units would cost **$195.00**. The mou' 6. -- got 'For slow-moving items, the usual goal is to avoid tying up c' -------------------------------------------------------------------------- model claude-opus-5 -------------------------------------------------------------------------- route-single (1 call per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. OK got 'auth' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. OK got 'billing' 10. OK got 'general' route-chain (3 calls per ticket) 1. OK got 'billing' 2. OK got 'billing' 3. MISS got '`auth`' 4. OK got 'bug' 5. OK got 'auth' 6. OK got 'bug' 7. OK got 'billing' 8. OK got 'bug' 9. FAIL got nothing [step 2: api error: BadRequestError] 10. FAIL got nothing [step 1: api error: BadRequestError] route-typed (1 call per ticket, messages.parse) STOPPED: 3 API errors in a row (last: BadRequestError: Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purcha). 38 calls made for this model; nothing sent after it. -------------------------------------------------------------------------- model claude-opus-4-6 -------------------------------------------------------------------------- route-single (1 call per ticket) 1. FAIL got nothing [api error: BadRequestError] 2. FAIL got nothing [api error: BadRequestError] STOPPED: 3 API errors in a row (last: BadRequestError: Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purcha). 3 calls made for this model; nothing sent after it. -------------------------------------------------------------------------- model claude-haiku-4-5-20251001 -------------------------------------------------------------------------- route-single (1 call per ticket) 1. FAIL got nothing [api error: BadRequestError] 2. FAIL got nothing [api error: BadRequestError] STOPPED: 3 API errors in a row (last: BadRequestError: Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purcha). 3 calls made for this model; nothing sent after it. ========================================================================== results ========================================================================== # model score single chain typed agent cost/run median s fail 1 claude-haiku-5-5 33/35 10/10 9/10 9/10 5/5 0.00360 0.802 0 2 claude-sonnet-5-5 33/35 10/10 9/10 9/10 5/5 0.08971 1.205 0 3 claude-opus-5-5 33/35 10/10 9/10 9/10 5/5 0.17514 1.379 0 4 claude-fable-5-2 33/35 10/10 9/10 9/10 5/5 0.33888 1.404 0 - claude-opus-5 17/20 10/10 7/10 0/0 0/0 0.20978 1.424 2 INCOMPLETE - claude-opus-4-6 0/2 0/2 0/0 0/0 0/0 0.00000 - 2 INCOMPLETE - claude-haiku-4-5-20251001 0/2 0/2 0/0 0/0 0/0 0.00000 - 2 INCOMPLETE - OpenAI UNMEASURED: no API key in this repository; not run and not estimated - Google UNMEASURED: no API key in this repository; not run and not estimated - Meta (Llama, open weights) UNMEASURED: no endpoint or key in this repository; not run and not estimated - Mistral AI UNMEASURED: no API key in this repository; not run and not estimated - DeepSeek UNMEASURED: no API key in this repository; not run and not estimated score: points out of the scored items (route-single 10, route-chain 10, route-typed 10, agent 5). cost/run: USD for this model's whole pass, logged tokens times the published price. median s: median wall time of one model call, from this runner. fail: items with no usable answer (API error, empty, max_tokens, raise or None, step cap). tokens, claude-haiku-5-5: in 14611, out 4280 (thinking 1391), 63 calls, stop reasons end_turn 56, tool_use 7 tokens, claude-sonnet-5-5: in 15767, out 5818 (thinking 1376), 63 calls, stop reasons end_turn 56, tool_use 7 tokens, claude-opus-5-5: in 14601, out 5837 (thinking 2073), 62 calls, stop reasons end_turn 56, tool_use 6 tokens, claude-fable-5-2: in 14265, out 5619 (thinking 2347), 62 calls, stop reasons end_turn 56, tool_use 6 failures, claude-opus-5: api error 2 tokens, claude-opus-5: in 6631, out 7065 (thinking not reported), 38 calls, stop reasons end_turn 35, None 3 failures, claude-opus-4-6: api error 2 tokens, claude-opus-4-6: in 0, out 0 (thinking not reported), 3 calls, stop reasons None 3 failures, claude-haiku-4-5-20251001: api error 2 tokens, claude-haiku-4-5-20251001: in 0, out 0 (thinking not reported), 3 calls, stop reasons None 3 finished (UTC) : 2026-11-20T15:21:32+00:00 wrote : /home/runner/work/_temp/out/runs/index-1-2026-11-20.jsonl wrote : /home/runner/work/_temp/out/runs/index-1-2026-11-20-results.json exit 3