Home / Blog / Benchmarks

Benchmarks

No MCP vs Finder vs GitNexus vs CodeGraph: code retrieval benchmark

Re-run on Finder 0.0.11 (October 4): without an MCP server, our simulated agent read about 148k tokens per question and found 29% of the right files. Finder returned about 2.3k tokens and found 93%, hitting the main file on 8 of 8 questions. GitNexus and CodeGraph answer in 0.16–0.21 s from a local graph but found 27% and 43%; Finder takes ~1.3 s on a never-seen question because it calls a remote embedder and reranker.

Published October 2, 2026 · Updated October 4, 2026 · 6 min read

Results

Updated October 4, 2026 for Finder 0.0.11 with GitNexus and CodeGraph re-indexed at the same commit. The October 2 run (Finder 0.0.10) had GitNexus at 0.34 / 5 of 8 and CodeGraph at 0.43 / 4 of 8 — within run-to-run noise of 8 questions. Finder latency is now reported cold (each question unseen): repeated questions hit Finder's 10-minute rerank cache and return in ~30 ms, which would not be a fair comparison.

Recall is the share of hand-checked ground-truth files that appear in the answer. "Tokens to read" is the response size divided by four. For the no-MCP setup it is the files the agent opened.

What "no MCP" costs

The baseline does what an agent does with only built-in tools. It greps once for each keyword in the question, then opens the eight files with the most matches, reading at most 2,000 lines of each (Claude Code's default Read limit). That is about 16 tool calls and roughly 60× more tokens than Finder's answer, and it still missed the main file on 6 of 8 questions. Keyword overlap and relevance are different things. Files like a large config or a central router mention everything, so they win the grep and crowd out the file that actually holds the logic.

Finder vs code-graph tools

GitNexus and CodeGraph are fast because they search a prebuilt symbol graph with lexical and structural ranking. When the question uses words that also appear in symbol names, as in P1 ("FCP/1 output format… citations"), both match or beat Finder: 0.8 recall against Finder's 0.6. When the question describes behavior in different words than the code uses, they miss. P4 asks about "semantic search ranking", but the code lives in vector/ and voyage_rerank.rs; GitNexus scored 0.0 and CodeGraph 0.4. P7 asks how the web UI "gets live indexing progress"; both scored 0. Finder embeds the code, so it matches meaning instead of names, and then expands through the call graph.

The trade-off is speed and precision. On a never-seen question Finder takes about 1.3 s instead of 0.16–0.21 s (two remote calls: embed and rerank; repeated questions are cached), and it returns more surrounding evidence (callers, tests, dependencies). That lowers precision (0.12 vs 0.16–0.17 for the graph tools) while raising recall.

Caveats, read these

• Home-field advantage. We wrote the questions and ground truth on Finder's own repository, and tuned Finder on them over seven rounds. Treat the gap as an upper bound.
• One tool per engine. We tested discovery only: GitNexus query and CodeGraph codegraph_explore. GitNexus's context, impact and detect_changes are strong at exact-symbol work, which this benchmark doesn't cover.
• Simulated baseline. The no-MCP setup is a deterministic grep-and-read script, not a model run. A real agent may search smarter, or read even more.
• Fresh indexes. GitNexus was re-indexed at the tested commit, and CodeGraph was indexed fresh (499 files in about 25 s).

The script, scripts/bench-mcp-vs-graph-tools.mjs, runs against any repo with your own questions. That is the fairest way to choose.

Which should you use?

• Unknown code, questions in plain words: Finder. Highest recall, and one call replaces a dozen greps and reads.
• You already know the symbol name: any graph tool, or plain grep, answers in milliseconds. Finder's own context and impact also run locally with no embedding call.
• Nothing installed: expect the agent to read a lot of code. In this test, about 60× the tokens.

Frequently asked questions