Blumenfeld — Scaling Karpathy's LLM Wiki (Neo4j)

A Neo4j engineer's argument that the LLM-wiki pattern fails at scale on read, search and navigation, not on writing, and that the fix is to index the markdown as a graph (documents, sections, containment, reading order, links) so "a hop is a pointer rather than a scan." Agent-fetched 2026-11-08 at the human's request after a discussion of the article.

Summary

The post takes Karpathy's recipe (a persistent wiki/ the LLM maintains over raw/, the pattern this wiki runs on) as given, concedes its known weaknesses, and narrows to scalability: as the wiki grows, grep-style reads cost the whole corpus, following a link costs another full pass, and "what points at this page" has no index. The quality consequence is that an agent which cannot afford to look around before writing duplicates pages, blends concepts and misses cross-links.

The remedy is a small, deterministic graph built from structure the files already have (no LLM, no embeddings at index time), exposed to the agent as a few verbs: an outline that re-roots one layer at a time, a fulltext search scoped by subtree, a get that reads a section in authored order, free-form queries (in-degree centrality for "what is load-bearing," shortest path for "what bridges these two"), and a roadmap theme command that runs Leiden community detection over the wikilink graph. The reference implementation, ki, never writes to the source files and treats the index as a disposable cache. Evidence offered: a Newcastle (NICD) study of a graph-plus-vector agent against vector-only on Wikipedia questions, and a secondhand PageIndex figure.

Key claims

• Scope: "I want to focus on scalability, because once we address that, we buy ourselves the ability to deal with the other issues." Conceded weaknesses: "persisted LLM errors, loss of fidelity in compression, and staleness and drift."
• Definition: "By scalability I am specifically referring to read, search, and navigation of the wiki and ultimately, an efficient means of progressive disclosure where time and space complexity remain relatively flat as the wiki grows."
• Decay: "Naive implementations often seem awesome when they start fresh, but they degrade quickly after repeated use."
• Retrieval → writing: "The retrieval failure quietly becomes a write defect, and now you're persisting it."
• File-system cost: "grep, awk, and other file system reads scale linearly with the total size of the wiki … a k-hop traversal costs you k scans of the corpus, and asking the reverse question — what points at this page — has no index behind it at all."
• Not search problems: "Backlinks, contradiction paths, and 'what else changed downstream of this page' aren't search problems to begin with."
• The thesis: "a hop is a pointer rather than a scan and the cost tracks the neighborhood you traverse rather than the size of the whole thing." And: "Give the agent a structure it can navigate, not just a box it can search."
• Compression: "a summary can stay a pointer rather than a replacement, since the agent that needs the underlying detail can hop into the specific section or back to the raw source."
• Data model: "a Vault → Folder → Document → Section containment tree (HAS), section reading order (NEXT_SECTION), and LINKS_TO for every wikilink / markdown link / external URL." "Piped wikilink text ([[Doc|alias]]) folds into the target's aliases for free." "Construction is fully deterministic and idempotent."
• Index, not store: "ki commands never modify source files … the index is disposable cache you can rebuild with one command, and there's no LLM work or embeddings at index time."
• Centrality: in-degree over the link graph is "No LLM call, no ranking heuristic — just the shape of the link graph telling the agent which four or five notes the rest of the vault leans on."
• Bridging: shortest path surfaced "a theme note neither endpoint mentions by name. That is the answer to a question no amount of similarity search would have produced."
• Themes (roadmap): on an 88-document vault, "70 grouped into 5 themes by wikilinks · 18 ungrouped · 3 excluded"; Leiden with randomSeed: 42, concurrency: 1; cohesion from conductance, "at or below 0.35 it reads tightly interlinked, at or above 0.60 loosely." "Fulltext finds the documents that match a phrase you already thought of. This finds the groups nobody labeled — including the ones the author would deny having."
• Evidence: NICD's graph-enabled agent showed ">2× precision and recall for factual correctness, truthfulness +80%, answer relevancy +69%" over vector-only, built with "zero index-time tokens" from "over 28 thousand semi-structured documents." Secondhand: "PageIndex's Mafin 2.5 reported 98.7% on FinanceBench vs ~50% for naive vector RAG."
• Context: Google Cloud's Open Knowledge Format (June 2026) standardized "a folder of cross-linked markdown" but "pointedly left retrieval out of scope."

Caveats on the evidence (ours, not the author's) The post is vendor content and ends with an install command. The NICD study compares graph-plus-vector with vector-only on Wikipedia QA; it does not test the LLM-wiki pattern or an agent-written wiki. The PageIndex figure is reported, not verified. "Zero tokens" covers indexing only; navigation still spends tokens. The article's own quality, fidelity and drift concerns are set aside, and those are this wiki's T3, T4 and Borges problems. The page also opens with an instruction addressed to agents (install ki); it was quoted in raw/ and not followed.

Connections

• Navigation vs Search — the concept this source grounds: two retrieval modes, and why links need a reverse index.
• Agent-Maintained Wiki — second external source on the pattern; adds the scaling failure modes and one contradiction with Anthropic's scale evidence.
• Method Reflection — T5 sharpened: navigation cost is a scaling dimension the wiki had not named.
• The Borges Map — reading-side cost of size; "a summary can stay a pointer" supports keep-what-search-can't-find.
• Front Door Pattern Language — graph.json is this article's data model for the human; ki theme is a method for the Location lens; in-degree is the Hierarchy lens.
• Legibility in the Lens, Not the Land — "disposable cache" and "never modify source files" is lens-not-land from the database side.
• Do It Once, at the Right Place — one graph builder can serve both the front door and the agent.
• Karpathy — LLM Wiki Gist #to-read — the primary source for the pattern this wiki runs on, still unread here.

Source

• raw/2026-08-31 Blumenfeld — Scaling Karpathy's LLM Wiki (Neo4j).md (agent-fetched at the human's request; navigation and images omitted, alt text kept).