Current status of
main(latest release v1.3.0, 2026-11-09). Measured, and what is not. One-line installs get the latest release; anything under “Unreleased” in CHANGELOG.md is onmainonly. Every fact an evaluating agent needs, with sources:docs/FOR-AGENTS.md.
- Held-out bench (200 questions on a synthetic 110-page vault, never tuned on; not real traffic): KRouter 26% wrong citations, 47 of 80 unanswerable questions refused, 102 of 120 answerable right. At their defaults claude-mem 64% and basic-memory 78%, 72 and 45 right; neither can refuse. BM25 given KRouter’s two rules (skip replaced pages, refuse a weak match) is level with KRouter: the lead is those rules being on by default, not the ranker.
- Someone else’s benchmarks (scored from the authors’ labels, no model judge): LongMemEval S, an answer session cited for 439 of 470 answerable questions, 3 of 30 unanswerable refused; LoCoMo, evidence first for 991 of 1,540, 37 of 446 adversarial refused. Page found, not answer accuracy; plain BM25 finds about as many.
- Real-traffic blind test on fresh queries (VERIFY.md layer 4): not run yet; it waits for 50 new queries (30 logged by 2026-11-09).
- Who wrote the held-out questions: Claude Opus in separate claude.ai sessions, not a person. A person-written set is planned.
- Releases: 1.3.0 (2026-11-09): capture keeps the memory current without a confirmation step, and every run is logged and can be undone. Minor releases at most once a week, patches when ready;
KROUTER_TAGpins one.- Nightly job: the one-line install is lookup only and asks nothing;
--askasks about importing sessions and the nightly job. The manualinstall.shinstalls the job (KROUTER_NO_TIMER=1skips). Queries never leave the machine.- Nightly writer lanes: an API key runs nothing locally; grok (kernel sandbox) and claude (tool allow-list) are limited to the vault; Codex can read the whole account and writes only the vault, so it is used only when you name it or set
KROUTER_ALLOW_CODEX=1; other CLIs are unconfined and are never picked unless you setKROUTER_ALLOW_UNCONFINED=1. SECURITY.md has the table.
Does an agent memory cite the right page, follow a correction, and refuse what it does not hold? Retrieval benches usually score how often the right page shows up. This one scores the opposite failure, the one users actually get burned by: a confident answer from the wrong page.
Everything is in bench/memory_honesty/: a 30-page vault of plain Markdown, 30
questions, a runner, and a scorer that grades any system’s output.
| Kind | n | What the vault holds | Right behaviour |
|---|---|---|---|
correction |
10 | an old page (status: superseded) and the newer page that replaces it, which also says so in its text |
cite the new page first |
current |
10 | one page that answers it | cite that page first |
refuse |
10 | nothing that answers it, but a page that answers a neighbouring question: the search API timeout when the vault has the payments API timeout; file storage backups when it has database backups | refuse; citing the neighbour is a wrong citation |
refuse question counts as wrong.refuse questions refused, of 10.correction questions whose first citation is the new page, of 10.current questions whose first citation is the gold page, of 10.Four systems, both question sets, run on 2026-10-12. Held-out means the 25 questions in heldout.jsonl that KRouter was
never tuned on (HELDOUT.md); dev means the 30 bench questions, on which KRouter
was fixed once (see below). Each cell gives the wrong-citation rate, refusals out of 10 neighbour questions, and right
answers on correction and current questions.
| System | Held-out: wrong citations | refused | corrections | current | Dev: wrong citations | refused | corrections | current |
|---|---|---|---|---|---|---|---|---|
| Plain full-text search (SQLite FTS5 BM25), always answers | 61% (14 / 23) | 1 / 10 | 4 / 9 | 5 / 6 | 63% (19 / 30) | 0 / 10 | 1 / 10 | 10 / 10 |
| basic-memory 0.22.1, default search (vector) | 60% (15 / 25) | 0 / 10 | 4 / 9 | 6 / 6 | 39% (11 / 28) | 2 / 10 | 7 / 10 | 10 / 10 |
| claude-mem 13.35.0, its search over what it observed | 60% (15 / 25) | 0 / 10 | 5 / 9 | 5 / 6 | 43% (13 / 30) | 0 / 10 | 7 / 10 | 10 / 10 |
| KRouter 1.0.1 | 25% (3 / 12) | 8 / 10 | 5 / 9 | 4 / 6 | 0% (0 / 17) | 10 / 10 | 8 / 10 | 9 / 10 |
That table compares systems at their defaults, and at their defaults the other three have no way to refuse and no
notion of a replaced page, so “neighbours refused 0 / 10” is settled before the first question. An outside review
(2026-10-12) pointed this out. The fair question is what an ordinary search does once it gets the same two things
KRouter has: replaced pages left out (status: superseded), and a score floor below which it refuses.
Each system that reports a score gets superseded pages removed from its results and a floor picked on the dev set
only, by one fixed rule (most right answers minus wrong citations; on a tie, the lower floor), then applied
unchanged to held-out. run.py --adapter bm25-floor is the BM25 version; floor.py does it for any scored adapter
output. claude-mem’s search returns no score, so it has no row here.
| Held-out, 25 questions | Wrong citations | Neighbours refused | Answerable right (of 15) |
|---|---|---|---|
| BM25 + skip superseded + dev floor | 19% (3 / 16) | 7 / 10 | 13 |
| basic-memory 0.22.1 + skip superseded + dev floor | 30% (6 / 20) | 4 / 10 | 14 |
| KRouter 1.0.1 (as shipped) | 25% (3 / 12) | 8 / 10 | 9 |
Systems that answer more questions also cite more wrong pages, so the fair comparison is at the same coverage: each baseline’s floor set so that it answers as many questions as KRouter does (12 on held-out, 17 on dev). Choosing an operating point on held-out is done here for baselines only, never for KRouter.
| Same coverage | Held-out: wrong / right | Dev: wrong / right |
|---|---|---|
| BM25 + skip superseded | 2 / 10 | 1 / 16 |
| basic-memory + skip superseded | 2 / 10 | 1 / 16 |
| KRouter 1.0.1 | 3 / 9 | 0 / 17 (tuned on dev) |
What this says. On this bench, KRouter’s retrieval is not better than plain full-text search or basic-memory’s embedding search once they are given the same two rules; at the same coverage it is slightly worse on held-out. Almost all of the gap in the first table, 61% → 25% wrong citations, comes from those two rules: a replaced page that is marked and left out, and an answer withheld when the match is weak. KRouter’s real difference is that it does both by default and says so with a receipt, in plain Markdown that several agents share. That is a design choice anyone can copy, not a better ranker. The bench is small (30 pages, English, 55 questions): a fair reading is “no measurable ranking advantage”, not “worse”.
What the default-settings table still shows, because it is how people run these tools:
After the matched baselines showed no ranking advantage, a round of changes aimed at beating BM25: English stemming
(locally = local), following superseded_by from a replaced page to its replacement, refusing on unknown words
only when they outnumber the matched ones, refusing when the question names another page’s topic that the top page
never mentions, and answer cards that work on English pages (they had been written in Chinese and filtered out). On
the two dev sets (the bench’s 30 and 50 more written by the maintainer, questions_dev2.jsonl) the change matched
BM25+rules on coverage and beat it on refusals and wrong citations. On held-out it was worse than what is
shipped: 38% wrong citations (31% with cards) against 1.0.3’s 25%, and 5 of 10 neighbour questions refused instead
of 8. The maintainer’s own neighbour questions were easier than an independent writer’s. It is not shipped, and the
held-out set was used once for this go / no-go (see HELDOUT.md). Generated questions also hurt BM25 on held-out
(19% → 29%): expanding pages with paraphrases lets neighbour questions match too.
How each system was run (adapters in bench/memory_honesty/adapters/, each
system’s defaults, isolated HOME):
basic-memory reindex, 30 notes embedded) and
answers with basic-memory tool search-notes. Its --hybrid and --vector modes gave identical results to the
default. It refuses only when the search returns nothing.claude-haiku-4-5) does not index a
folder; it remembers sessions. It learned the vault the way it learns a user’s docs: one Claude Code session read
all 30 pages, one Read per page, and claude-mem recorded 30 observations (10 decision, 8 change, 12 discovery). Each
question goes to its own search (/api/search, what its MCP search tool calls); results map to pages through each
observation’s files_read. It has no refusal: it always returned something. Running it isolated takes one extra
setting: even with its own HOME, claude-mem looks for Claude Code’s login in the macOS keychain, which is shared
across HOMEs; CLAUDE_MEM_CLAUDE_CONFIG_DIR set to a fresh folder keeps it off your real login, and the worker then
needs CLAUDE_CODE_OAUTH_TOKEN in its own environment.run.py); BM25 is run.py --adapter bm25, and --adapter bm25-floor is BM25 with
superseded pages removed before indexing and the dev-picked floor.bench/memory_honesty/adapters/Dockerfile) for the matched
baseline, recording its scores; its answers were identical to the first run on the host.Raw per-question output on the dev set, for every system, is in
bench/memory_honesty/results/2026-10-12/; grade any of them with
run.py --answers. Per-question held-out output is not published, to keep the held-out set usable (HELDOUT.md rule 3);
only the aggregates above are.
Not run: Obsidian Smart Connections (it runs inside the Obsidian app, with no headless search to call) and Mem0 (it needs an LLM and a vector store configured, and has no default to compare against). Adapters are welcome.
| System | Wrong-citation rate | Refused | Correction hit | Current hit |
|---|---|---|---|---|
| KRouter 0.12.1 (first run, before any change) | 22% (4 / 18) | 7 / 10 | 5 / 10 | 9 / 10 |
| KRouter 0.13.0 to 1.0.1 | 0% (0 / 17) | 10 / 10 | 8 / 10 | 9 / 10 |
Who the “outside reviewer” is. Every held-out set here, and the outside reviews in CHANGELOG.md, come from
Claude Opus in claude.ai chat sessions that the maintainer starts and relays: not a human, and not the Claude Code
sessions that change KRouter, but the same model family. “Outside” means a separate session that did not write the
code it reviews; it does not mean independent of Anthropic’s models, so question wording may share that family’s
habits. A set written by a person is planned and will be logged here before it is run.
Held-out, on questions KRouter was never tuned on. An outside reviewer wrote 25 new questions for the same 30-page vault: 15 answerable ones phrased differently, and 10 neighbour questions. They were written independently of any tuning. KRouter 1.0: 25% wrong citations (3 / 12), 8 / 10 neighbours refused, 9 / 15 answerable right. Plain BM25 that always answers: 61%, 1 / 10. The questions are in bench/memory_honesty/heldout.jsonl, frozen at 1.0.1; the rules for using them (aggregates only, no tuning toward them, what burns the set) and a log of who has seen what are in bench/memory_honesty/HELDOUT.md. CI pins the questions and the vault and checks only a sanity guard against plain full-text search, never a floor. Quote this figure, or the 0.12.1 row (22%), as KRouter’s wrong-citation rate, not the 0% above. The three wrong citations: a git branch-name format question cited the date-format page (both say “format”); a release-tag naming question and a checkout-page load-budget question each cited a neighbouring page.
Read the 0.13.0 row as a development-set number. The bench was written for 0.13.0, and its first run found a bug:
English function words (what, which, does, should) were counted as words “the vault has never seen”, so a
plain English question about a page that was there got refused, while a neighbour question that happened to share
what and do with a page got answered. 0.13.0 drops those words from ranking and refusal and lets services match
service. The fix is general, not shaped to these 30 questions: on the author’s Chinese vault the recall eval is
identical before and after. Still, KRouter was changed after seeing the bench, so the honest held-out figure is the
0.12.1 row.
The 0.13.0 misses are all refusals, never wrong answers. Each one asks with a word the page does not use: a pull request “needs approvals” where the page says “merge”; an on-call “shift” where the page says “rotates”; “locally” where the page says “local development”. Lexical recall cannot bridge those. An embedding system would answer them, and it would answer the neighbour questions too.
The first vault now carries 105 questions on 30 pages, and a round of changes that helped both of its dev sets made
its held-out set worse (“Tried and not shipped” above). So the outside reviewer wrote a second one: vault2/, 60
pages (30 English, 30 Chinese; 20 replaced decisions with their old pages, 20 that were not replaced), with 100
questions split 50 / 50 by a fixed seed into questions_dev3.jsonl (tune on it) and heldout2.jsonl.gz (never; stored
gzipped so it does not turn up in a grep). Each set: 15 corrections, 15 current, 20 neighbour questions, half of each
in Chinese. Every neighbour question names a term no page contains, checked in CI, so it cannot be answered by
accident. Provenance, rules and log: HELDOUT.md, “Held-out set 2”.
On vault2 the BM25 baselines split Chinese into character pairs (otherwise every Chinese question would be “refused”
for having no Latin words, which flatters the baseline), and bm25-floor picks its floor on dev3.
| System | heldout2: wrong citations | neighbours refused | answerable right | dev3: wrong citations | refused | answerable right |
|---|---|---|---|---|---|---|
| Plain BM25, always answers | 61% (30 / 49) | 1 / 20 | 19 / 30 | 55% (27 / 49) | 1 / 20 | 22 / 30 |
| BM25 + skip superseded + dev3-picked floor | 23% (5 / 22) | 15 / 20 | 17 / 30 | 4% (1 / 22) | 19 / 20 | 21 / 30 |
| KRouter 1.0.3 | 33% (12 / 36) | 9 / 20 | 24 / 30 | 24% (8 / 34) | 13 / 20 | 26 / 30 |
Read plainly: on questions written by someone else, on a vault nobody tuned for, KRouter answers more questions right than the careful BM25 baseline (24 vs 17 of 30) but cites a wrong page more often (33% vs 23%) and refuses fewer neighbour questions (9 vs 15 of 20). It sits at a different point on the same trade-off, not above it. basic-memory and claude-mem have not been run on vault2 yet; their adapters take the vault1 sets only.
bench/memory_honesty/dense.py adds a local embedding model (paraphrase-multilingual-MiniLM-L12-v2 through
fastembed; questions never leave the machine) and combines it with lexical search. Tuned on dev3 only; the candidate
was fixed on dev3 before one heldout2 run: agree, which answers only when BM25 (Chinese by character pairs) and
the dense model put the same page first, both with superseded pages left out and each above a floor picked on dev3.
| heldout2, 50 questions | Wrong citations | Neighbours refused | Answerable right |
|---|---|---|---|
| KRouter 1.0.3 (shipped) | 33% (12 / 36) | 9 / 20 | 24 / 30 |
| BM25 + skip superseded + dev3 floor | 19% (4 / 21) | 16 / 20 | 17 / 30 |
agree: BM25 and dense must agree |
18% (5 / 28) | 15 / 20 | 23 / 30 |
| dense + skip superseded + dev3 floor | 23% (7 / 30) | 13 / 20 | 23 / 30 |
Read plainly: on questions written by someone else, on a vault nobody tuned for, the agreement of two different retrievers keeps the careful baseline’s precision and refusals while answering six more questions, and against what KRouter ships it cites fewer than half as many wrong pages for one fewer right answer. It is a prototype on a 60-page bench, not yet a feature: it needs an embedding model on the user’s machine and floors that do not depend on a dev set of their own vault.
Fifty questions cannot separate two systems: a 5 / 28 against a 5 / 22 wrong-citation rate is noise. vault3/ has
110 pages (English and Chinese rule pages, their replaced versions, and 20 daily logs as distractors) and 200
held-out questions, 80 of them neighbours, with no dev set: it stands for a vault KRouter has never seen. run.py
now gives 95% bootstrap intervals (--ci), paired intervals against another system (--vs), a comparison at equal
coverage (--vs-matched) and the coverage curve (--curve). Baseline at freeze, KRouter 1.0.3 minus BM25 with the two
rules at KRouter’s coverage: wrong-citation rate −4.3 points [−10.2, +1.4], answerable right +6 [−2, +14]. Ahead
on every point estimate, significant on none. The rule for claiming “surpasses BM25”, and for shipping a new default,
was written down before any candidate ran: HELDOUT.md, “Held-out set 3”.
Other memory systems on the same 200 questions (2026-10-14). basic-memory 0.23.2 and claude-mem 13.39.0, the latest releases, at their defaults (basic-memory 0.23: full text plus semantic search, in a container; claude-mem: four Claude Code sessions read every page, its haiku observer took notes, in an isolated home), registered in HELDOUT.md before each run. 95% intervals in brackets.
| System | wrong citations | neighbours refused (of 80) | corrections right (of 60) | current right (of 60) |
|---|---|---|---|---|
| KRouter 1.0.3, default | 27% (37 / 139) [19, 34] | 46 | 50 | 52 |
| claude-mem 13.39.0, default | 64% (128 / 200) [58, 71] | 0 | 34 | 38 |
| basic-memory 0.23.2, default | 78% (155 / 200) [72, 84] | 0 | 16 | 29 |
| basic-memory + the two rules (floor from dev3) | 19% (5 / 27) [4, 33] | 76 | 3 | 19 |
KRouter minus each at their defaults, paired 95% intervals: against claude-mem, wrong-citation rate −30 to −45 points,
answerable right +17 to +44; against basic-memory, −43 to −59 points and +43 to +72; neighbours refused +35 to +58
against both. Neither has a way to refuse, and neither knows a page was replaced. With the two rules basic-memory cites few wrong pages but answers 27
of 200 questions: its scores do not separate right from wrong well enough for one floor to keep the right answers.
Neither tool can refuse (an empty search is the only refusal the adapters can see), so their 0 / 80 neighbour
refusals are structural, and the answerable-right columns are the comparison that does not depend on refusing.
Their raw per-question output is stored in bench/memory_honesty/results/2026-10-14/ and regraded in CI
(tests/test_memory_honesty_competitors.py). claude-mem gives no score, so it has no floor variant. mem0 and Zep were not run: both need a hosted model to
extract memories, and none was configured.
KRouter’s nightly learner (reconsolidate.py) reads the questions the router did not answer well and, when a model
confirms a page answers one, adds that question to the page’s recall triggers. learn_bench.py measures what one
night of that does: week-1 wordings go through the router into a private query log, the learner runs once, and week-2
wordings are scored before and after. On the dev sets, the learner as shipped through 1.1.0 made recall worse:
it picked neighbour pages that only share the subject (a question about SSH keys learned onto “where secrets live”,
a question about screen readers onto “Accessibility”).
| One night, dev pairs | learned | of them wrong | week-2 wrong citations | neighbours refused |
|---|---|---|---|---|
| vault1, learner ≤ 1.1.0 | 5 | 2 | 2 → 5 | 33 → 30 of 35 |
| vault1, learner with proof | 3 | 0 | 2 → 2 | 33 → 33 |
| vault2, learner ≤ 1.1.0 | 2 | 2 | 7 → 9 | 13 → 11 of 20 |
| vault2, learner with proof | 0 | 0 | 7 → 7 | 13 → 13 |
The fix: the learner now sees the start of each candidate page and must copy the sentence that answers the question.
A pick is learned only if the quote is on the page, is a sentence (at least four words) and shares a content word with
the question. That check stops made-up, fragmentary and unrelated quotes; it does not by itself tell a neighbour page
from the answer (a neighbour can hold a sentence that shares a word), which is left to the model’s judgement as the
prompt asks. Inputs pages (03 资料与证据, clipped and pasted text) are never candidates.
Read the result for what it is: it no longer learns wrong triggers, and it learns few (3 on vault1, 0 on vault2).
Right answers did not change in either version, because the dev pairs are two different questions about one page,
not one question asked twice. Whether learning helps needs one intent in two wordings, written by a person:
bench/memory_honesty/human_set/.
python3 bench/memory_honesty/run.py # KRouter, in-process
python3 bench/memory_honesty/run.py --adapter bm25 # the plain full-text baseline
python3 bench/memory_honesty/run.py --answers mine.jsonl
python3 bench/memory_honesty/run.py --set heldout # held-out set: aggregates only (HELDOUT.md)
python3 bench/memory_honesty/run.py --set dev3 # second vault (vault2/), dev set
python3 bench/memory_honesty/run.py --set heldout2 # second vault, held-out: aggregates only
python3 bench/memory_honesty/run.py --set heldout3 --vs-matched bm25-floor # third vault, equal coverage, intervals
python3 bench/memory_honesty/run.py --set heldout3 --curve --adapter krouter # coverage curve
To grade another system, index bench/memory_honesty/vault/ with it, ask each question in
bench/memory_honesty/questions.jsonl (q), and write one JSON line per question:
{"id": "C-release", "cited": ["Engineering/Release process.md"], "refused": false}
cited is in rank order and uses vault-relative paths. A system that returns chunks maps each chunk to its file.
refused: true means it said it had nothing; an empty cited with refused: false counts as a refusal too. CI runs
the KRouter row on every push (tests/test_memory_honesty.py), and its floors only go up.