Verdict
Quality matched on 100% of cases and the candidate is cheaper. Latency is informational during calibration.
Cases evaluated
Evaluation complete
Quality matches
100%
Est. monthly savings
Projected from matching events observed over 30 days
Status
The experiment finished and its verdict is available.
Controlled replay compares both models from the same worker. Production latency is shown separately as context and is not subtracted from candidate replay latency.
claude-sonnet-4-6 and claude-haiku-4-5 received the same frozen prompts in alternating order.
Replayed source p50
1,419 ms
claude-sonnet-4-6
Replayed candidate p50
626 ms
claude-haiku-4-5
Paired median delta
-762 ms (-55%)
95% interval: -874 ms to -690 ms
Controlled p90
1,624 ms / 735 ms
Source / candidate
Production source context
1,027 ms p50, 2,905 ms p90
30 captured measurements from the user's environment
Controlled coverage
30 successful pairs
Only complete source/candidate pairs enter the comparison (provider-completion-v1)
The evaluation compares structured content canonically and reports presentation adherence as separate evidence unless strict formatting is enabled for the feature.
This run's encrypted evidence expires at the date shown above, 14 days after the run was created. The verdict summary and savings estimate remain in experiment history after that run-specific evidence is deleted.
Review the case results before opening sensitive prompt and response evidence. Opening a case is audited.
| Case | Result | Format | Evaluation confidence | Attempts | Production source | Replayed source | Candidate | Replay delta | Status | Case details |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Equivalent | Met | 97% | 2 | 622 ms 35,383 in / 1,608 out | 1,200 ms 35,383 in / 1,608 out | 510 ms 33,968 in / 1,351 out | -690 ms (-57%) | Evaluated |
Replay failures
0 source, 0 candidate
Failures are excluded from percentiles and retained as reliability evidence
| View details |
| 2 | Equivalent | Met | 97% | 1 | 1,502 ms 44,229 in / 1,832 out | 1,247 ms 44,229 in / 1,832 out | 541 ms 42,460 in / 1,539 out | -706 ms (-57%) | Evaluated | View details |
| 3 | Equivalent | Met | 97% | 1 | 1,086 ms 103,496 in / 4,762 out | 1,294 ms 103,496 in / 4,762 out | 572 ms 99,356 in / 4,000 out | -722 ms (-56%) | Evaluated | Hide details |
Case 3 evidenceFull captured evidence for this scored comparison. EquivalentFormat metscored97% evaluation confidenceJudge: gemini-3.1-pro-preview Evaluation notesThe candidate preserved the original answer's policy, resolution, and customer-safe guidance. Format evidenceThe response preserved the expected support-answer structure. Captured prompt | ||||||||||
| 4 | Equivalent | Met | 97% | 1 | 478 ms 37,152 in / 1,921 out | 1,341 ms 37,152 in / 1,921 out | 603 ms 35,666 in / 1,614 out | -738 ms (-55%) | Evaluated | View details |
| 5 | Equivalent | Met | 97% | 1 | 882 ms 43,345 in / 1,653 out | 1,388 ms 43,345 in / 1,653 out | 634 ms 41,611 in / 1,389 out | -754 ms (-54%) | Evaluated | View details |
| 6 | Equivalent | Met | 97% | 1 | 2,845 ms 101,196 in / 4,298 out | 1,435 ms 101,196 in / 4,298 out | 665 ms 97,148 in / 3,610 out | -770 ms (-54%) | Evaluated | View details |
| 7 | Equivalent | Met | 97% | 1 | 1,623 ms 221,146 in / 8,265 out | 1,482 ms 221,146 in / 8,265 out | 696 ms 212,300 in / 6,943 out | -786 ms (-53%) | Evaluated | View details |
| 8 | Equivalent | Met | 97% | 1 | 797 ms 108,096 in / 4,530 out | 1,529 ms 108,096 in / 4,530 out | 727 ms 103,772 in / 3,805 out | -802 ms (-52%) | Evaluated | View details |
| 9 | Equivalent | Met | 97% | 1 | 261 ms 42,460 in / 1,876 out | 1,576 ms 42,460 in / 1,876 out | 758 ms 40,762 in / 1,576 out | -818 ms (-52%) | Evaluated | View details |
| 10 | Equivalent | Met | 97% | 1 | 1,605 ms 98,896 in / 4,879 out | 1,623 ms 98,896 in / 4,879 out | 529 ms 94,940 in / 4,098 out | -1,094 ms (-67%) | Evaluated | View details |
Northstar support case 3: resolve the ticket-summary request with the account policy and cited help-center context.
claude-sonnet-4-6
Resolved ticket-summary case 3 with the correct policy, next step, and customer-safe explanation.
claude-haiku-4-5
Cheaper candidate preserved the policy, action, and explanation for ticket-summary case 3.