Home / Blog / Benchmarks

Benchmarks

Finder 0.0.10 benchmark: 0.95 recall at ~770 ms

On 8 ground-truth questions over a real Rust codebase, Finder 0.0.10 found 0.95 of the expected locations with a 767 ms median, versus 0.53 recall at 3.7 s for a hosted context-engine MCP server. It trails on precision and on top-1 file ranges — details below.

Published October 1, 2026 · 5 min read

Method

Apple M1 Pro, 10 cores. Repository: the Finder codebase itself (Rust + TypeScript). Sequential MCP calls, engines interleaved per run, one untimed prewarm per query, then three timed runs. Each question has hand-checked ground-truth locations; recall is the share of those locations returned, "primary hit" means the main location appeared at all.

Query retrieval

Finder 0.0.10 is about 4.8× faster at the median and finds nearly twice as many expected locations. The hosted engine returns tighter result sets (higher precision), but misses the main location on 3 of 8 questions — and a missed location is something the agent then has to find by reading files.

File retrieval

Here Finder loses on ranking: the right range is always in the top 5, but first only 62% of the time. Improving hit@1 toward 0.85 with tighter ranges is the next item on our roadmap.

Takeaways

• For discovery questions, recall matters most: a missed location costs far more tokens than a few extra candidates.
• Latency under a second keeps agents from timing out or falling back to grep.
• Eight questions on one repo is a small sample. We publish the raw numbers per release so you can follow the trend.

Frequently asked questions