KeeFetch / Benchmarks

Evidence behind the profiles.

KeeFetch provider study methods, corpus, machine-review provenance, limitations and reproduction.

Expanded study complete

The v1.3 source candidate uses a fresh 19-candidate comparison, including the full chain without Yandex. All 133 cells and 827 reviewed icon units passed validation. Four unresolved app identities remain ambiguous and receive zero usability credit under the owner's conservative-scoring decision.

Corpus and method

The public v1 corpus contains 300 fixtures. The expanded experiment uses three repetitions, concurrency eight and cold/warm modes: 114 measured cells plus 19 unmeasured warm-ups. The experiment, corpus, plugin binary, execution harness and effective policies are fingerprinted.

Cold results determine winner metrics. Warm rows provide informational latency measurements. Provider calls, disclosure, usable/correct icons, errors and elapsed time are separate measures.

Every cold artifact is reviewed

The review unit is the exact fixture ID and artifact hash. The census includes every unique cold artifact and propagates its label to matching occurrences. Labels are correct, acceptable-synthetic, generic, wrong-brand, blank, unusable and ambiguous.

Base labeling comprises 560 agy Gemini 3.8 Flash labels and 267 owner-run ZCode GLM-5.3-Flash labels, with separate provenance. Pilots, focused re-asks and direct Codex corrections are disclosed. The final direct spot-check covers 138 units and 92 images. This is machine review; ZCode's model identity is harness-reported and its transcripts are reconstructions.

What the numbers cannot promise

Network timing, provider behavior and website icons change. A census removes sampling error for its population, not reviewer judgment error. Results apply to the measured corpus and candidates, not every site. Candidate names containing “thorough” are experiment identifiers; the user-facing profile is Precise.

Interrupted partial cells were quarantined and rerun; no resumed latency was accepted. Collection used a shared developer machine. The strict ambiguity replay changed Precise's winner, so the owner explicitly chose conservative failure scoring. Fast, Balanced and Privacy remained stable across replay scenarios; Precise did not. The two leading precision candidates are close, and this result does not establish broad superiority.

Inspect and reproduce

Evidence links below point to the completed study snapshot, not an older report on the default branch.

Full study report · Public corpus · Harness and selector

powershell -NoProfile -ExecutionPolicy Bypass -File eng/benchmark-presets.ps1 -Experiment eng/benchmark/experiments/profile-candidates-v13-complete.json

Use the matching source revision, a built plugin and KeePass 2.60. A live run contacts the public fixtures and enabled providers. Preserve fingerprints and complete the census before selecting winners. The accepted selection uses -AmbiguityPolicy ConservativeFailure; the default selector still requires stable winners.