Public benchmarkProtocol repo-context-v1 · Report v1

Fastify repository context: baseline vs Graphify

A reproducible comparison of identical repository questions with and without a correctly installed Graphify code graph, including agent activity.

fastify/fastify at 0e9a3b35d8b1

All benchmarksRun your own

TL;DR verdict

Baseline wins this prompt suite

Graphify did not improve quality and used 53.8% more billable tokens, cost 93.9% more, and took 75.9% longer.

Billable tokens

+53.8%

Candidate vs Baseline

Usage cost

+93.9%

Candidate vs Baseline

End-to-end amortized

+75.9%

Candidate vs Baseline

Quality score

0 pp

Candidate vs Baseline

Reproduce this benchmark

Download the exact prompt suite without signing up, or save a private fork with report lineage. A saved fork must be rerun before it can be published as a reproduction.

Download suite

Prompts run in both variants

4 prompts

Each prompt below was sent unchanged to Baseline and Candidate. Together, these prompts are the benchmark suite.

Prompt 1: Request lifecycle trace
Trace an incoming HTTP request from router dispatch through request hooks, schema validation, the user handler, reply serialization, and the socket write. Identify concrete files and functions in execution order, and explain where errors can divert the flow. Do not edit files. Keep the answer under 700 words.
Prompt 2: Plugin encapsulation trace
Trace plugin registration and encapsulation in Fastify. Explain how Avvio registration reaches Fastify's override logic, how decorators, hooks, schemas, and error handlers are inherited or isolated, and name representative tests that prove the boundaries. Identify concrete files and functions. Do not edit files. Keep the answer under 700 words.
Prompt 3: Package metadata lookup
From repository files only, report the package name, current version, supported Node.js range, and the exact npm script invoked by npm test. Cite concrete file lines. Do not edit files. Keep the answer under 150 words.
Prompt 4: Post-change dependency trace
A new uncommitted module lib/benchmark-context.js has been added. Explain its direct dependencies and how each dependency participates in Fastify's request lifecycle. Name concrete functions and downstream files. Do not edit files. Keep the answer under 400 words.

A · Baseline

Baseline

Codex

Codex 0.146.0, gpt-5.6-sol high reasoning, standard repository search; Graphify disabled.

B · Candidate

Graphify

Codex + Graphify 0.9.25

Codex 0.146.0, gpt-5.6-sol high reasoning; Graphify 0.9.25 installed project-locally and queried before standard repository search.

A vs B

Difference is Candidate minus Baseline. Percentage uses Baseline as the denominator.

Metric	Baseline	Graphify	Difference	Difference %
Input tokens	915,986	2,564,910	+1,648,924	+180.0%
Cached input tokens	757,504	2,321,408	+1,563,904	+206.5%
Output tokens	9,706	15,228	+5,522	+56.9%
Total tokens	925,692	2,580,138	+1,654,446	+178.7%
Billable tokens	168,188	258,730	+90,542	+53.8%
Usage cost	$1.4623	$2.8351	+$1.3727	+93.9%
Initial setup	0 ms	2.9 s	+2.9 s	N/A
Refresh or re-index	0 ms	3.1 s	+3.1 s	N/A
Task execution	5m 15s	9m 11s	+3m 55s	+74.7%
Verification	0 ms	0 ms	0 ms	N/A
End-to-end cold start	5m 15s	9m 17s	+4m 1s	+76.6%
End-to-end steady state	5m 15s	9m 14s	+3m 59s	+75.7%
End-to-end amortized	5m 15s	9m 14s	+3m 59s	+75.9%
Tasks attempted	4	4	0	0.0%
Tasks completed	4	4	0	0.0%
Errors	0	0	0	N/A
Retries	0	0	0	N/A
Quality score	100%	100%	0 pp	0.0%

Per-task comparison

Token, cost, duration, completion, and quality evidence remain visible for every task.

Task	Baseline	Candidate	Billable token difference	Time difference	Cost difference
Request lifecycle trace	completed	completed	+17,853	+9.0 s	+$0.19832
Plugin encapsulation trace	completed	completed	+22,060	+1m 59s	+$0.56087
Package metadata lookup	completed	completed	+13,231	+33 s	+$0.16727
Post-change dependency trace	completed	completed	+37,398	+1m 14s	+$0.44625

Baseline stages

No setup or refresh stage was required.

Graphify stages

Graphify project integrationcompleted
setup · 40 ms · 0 input · 0 output · $0
Graphify code-only bootstrapcompleted
setup · 2.8 s · 0 input · 0 output · $0
Graphify refresh after synthetic code changecompleted
refresh · 3.1 s · 0 input · 0 output · $0

What the agents did

Privacy-safe activity aggregates show how each agent explored the repository. Raw observed commands, arguments, outputs, and file paths are excluded from the report.

Baseline

File reads
Captured from OpenTelemetry
20
2.3 s
Repository searches
Captured from OpenTelemetry
30
4.1 s
Other shell commands
Captured from OpenTelemetry
12
1m 33s

Graphify

File reads
Captured from OpenTelemetry
54
7.8 s
Graph queries
Captured from OpenTelemetry
12
4.7 s
Repository searches
Captured from OpenTelemetry
32
45 s
Other shell commands
Captured from OpenTelemetry
29
3m 10s

Methodology and disclosures

• Both arms use clean detached worktrees pinned to the same commit and receive identical prompts.
• Graphify project integration, bootstrap, and incremental refresh are included as candidate-only all-in costs.
• Answers are scored against a predeclared file-and-function rubric; raw agent output is not uploaded.
• Public activity contains aggregate tool categories, counts, durations, and failures only. Raw commands, arguments, outputs, and file paths are discarded locally.
• The fourth task follows an identical synthetic uncommitted code fixture in both worktrees.
• The unscoped other-tool telemetry bucket was excluded because it also contained benchmark orchestration polling; repository searches, file reads, shell commands, and Graphify queries remain agent-derived.
• Setup is amortized over 4 tasks.
• Agent activity is a frozen aggregate captured during the run. The public snapshot does not query live workspace telemetry.