PK Wiki

overview

Method Reflection

status: active · created 2026-11-08 · #moc #meta #method

The wiki looking at itself. Overview maps what the wiki knows; this page tracks how it knows, and where its sources argue that method is wrong, incomplete or working. It is a second home page: start here when you want to improve the wiki rather than read it.

Maintenance rule: on every INGEST, LINT or QUERY, the agent asks whether the material bears on how this wiki captures, stores or uses knowledge. If it does, it updates a tension below (or opens a new one) and logs it. Tensions are never silently resolved. They close only with a recorded verdict.

The method, in one paragraph

Lineage: the method is Karpathy's LLM-wiki recipe (April 2026), read as a primary source only on 2026-11-08. Two of its lines have since been revised here: the "10-15 wiki pages" per source became a quota and was dropped (T5), and "no accumulation" with RAG became "not a RAG system" and was qualified (T5). Its "stay involved" preference is the branch T4 left.

Sources go into raw/ once. The agent reads each one, integrates it into linked pages, and keeps a running synthesis, so knowledge accumulates instead of being re-derived per question. Claims carry provenance in three states:

  • active: backed by a source.
  • draft/#unsourced: an established fact the human relayed.
  • seed/#hypothesis: the human's hunch.

Hunches are promoted only when a source grounds them. A large share of the wiki (the engineering-craft domain) consists of distilled maxims: the human's rules of thumb, each turned into a concept page.

Live tensions

T1 — Maxims without knowledge open · opened 2026-11-08

Challenge. Willingham argues maxims about how to think are [[Metacognitive Strategies|cues, not capabilities]]. They tell you what to do, and the how comes from domain knowledge and practice (Critical Thinking Is Domain-Bound). Generic thinking programs that taught strategies across many problem types produced only modest, non-transferring gains.

Where it bites. The ~14 craft pages from Working Heuristics — Data, Matching, Product, Org, Tooling are exactly that kind of list. Read cold by someone without the data-engineering context, Willingham predicts they'll be memorized but not applied. Even for the human, the maxim helps only if it fires at the right moment.

What it doesn't say. For the human who wrote the heuristics, each one is a compressed pointer to real experience, so the knowledge is already there. The risk is to other readers, including future agents and teammates, and to the human after the context fades.

Candidate responses (not yet adopted; human's call):

  • Give each craft page a ## Worked examples section with at least two concrete cases from real work, following Willingham's recipe: two examples, a label, content at hand.
  • Add a LINT check: flag heuristic pages with fewer than 2 worked examples.
  • Treat a heuristic as mature only when it has fired in a new context and been logged. That is evidence of transfer, not just statement.

Sharpened 2026-11-08 (two more sources). Alexander — The Origins of Pattern Theory (OOPSLA 1996) makes the same critique from design: a collection of patterns is "fragmentary, atomic, ideas" that "contain a bunch of good ideas" but don't generate a coherent whole until they're ordered into a generative sequence. And the Borges check measured the cost: the craft cluster is 11.1× its source. Converging remedy (proposed): compress the 11 craft pages into fewer pages, each with ≥2 of the human's worked examples, ordered as a sequence.

T2 — Filing by surface, linking by depth affirming · opened 2026-11-08

Folders and domains (concepts/, "AI and work", "engineering craft") file knowledge by surface. Surface vs Deep Structure explains why cross-domain links are hard to see and why they're the valuable part. That supports the wiki's existing bet on dense linking and on cross-domain syntheses such as Placement and Judgment Over Raw Execution and Do It Once, at the Right Place, which are explicit deep-structure claims. Implication: syntheses are not optional extras. They're how the wiki does what memory does badly by default.

Grounded 2026-11-08. Alexander — A City Is Not a Tree gives T2 its formal shape: folders are a tree, links a semilattice, and "in a single mental act you can only visualize a tree" (Tree vs Semilattice). He also adds a limit: "a garbage can is full of overlap," so links must be the right overlap (The Borges Map).

T3 — Who holds the knowledge? open · opened 2026-11-08

The agent does the reading and writing; the human curates and asks. Willingham says critical thinking runs on knowledge held in the thinker's head, not on a page. So an agent-maintained wiki could either:

  • build the human's judgment, by giving them dense, linked material to think with, or
  • substitute for it, as a lookup that lets the knowledge never move into the head.

Karpathy states the division as a design goal: "The human's job is to curate sources, direct the analysis, ask good questions, and think about what it all means. The LLM's job is everything else" (Karpathy — LLM Wiki Gist). T3 is the question that goal leaves open. That is the wiki-scale version of the open question on Critical Thinking Is Domain-Bound (does offloading knowledge to AI erode judgment?), and of the "judgment is the human premium" claim in What's Scarce When Execution Is Cheap.

Signals worth watching: INGEST's "discuss before writing" step and the "teach as we go" practice (Metacognitive Strategies) are the wiki's current counterweights, because both force the human to engage with the content. Automating ingest would remove one of them.

Sharpened (2026-11-09, daily pass). Ashby — An Introduction to Cybernetics gives T3 a measure. Regulation is communication: "R's capacity as a regulator cannot exceed R's capacity as a channel of communication." The human regulates the agent's output through a channel (a PR summary, a discuss-before-writing step, the daily list), and the law says the regulation can be no finer than that channel's capacity. Shrinking the channel to save effort (unattended ingest, T4) shrinks the control with it. That is the mechanism behind "review degrades toward approval."

Sharpened (auto-ingest 2026-11-08). Anthropic — How Scientists Use Claude shows how working scientists split the labour on their own. AI takes data analysis (33.7% of chat sessions) and writing (32.0%). Ideation and hypothesis generation stay human (4.2%), even though scientists rank ideation most important. That is exactly this wiki's split: the agent reads, synthesizes and writes, and the human chooses the questions. On that evidence the "build" branch survives only if the human keeps doing the ideation (asking questions, capturing seeds, judging tensions) instead of only approving output. The same survey has 72% expecting AI to match top researchers at ideation by 2030, so this division may be temporary. If the agent also starts proposing the questions, the wiki moves further toward "substitute."

Sharpened again (auto-ingest 2026-11-08, second source). Anthropic — Investigating Vault Particles with Claude turns T3 from a prediction into a priced observation. Anthropic's own biology team worked outside its expertise on purpose, and found: "In areas we knew well, we caught mistakes quickly, but they cost us time in areas we didn't." Two weeks lost to one flawed protocol; a mirror-image cryo-EM map that stood because nobody present could see it was wrong (Verification Requires Domain Knowledge).

Read onto this wiki: the human reviews the agent's output. Where the human already holds the domain, review catches errors cheaply. Where they don't — which is most of why the wiki exists — review degrades toward approval. That is the "substitute" branch arriving not by erosion but by default, and T4's auto-ingest PR queue is exactly such a review surface.

What the case suggests as counterweights (the human's call, none adopted):

  • External ground truth beats internal coherence. The lab's errors were ultimately caught by gels and maps. The wiki's equivalent is raw/ and web search — which also argues for the wiki-plus-search finding in T5 rather than wiki-only review.
  • "Do normal things unless they don't work." A convention-first default is a cheap substitute for expertise. The wiki's version would be: prefer the source's own framing and vocabulary over the agent's reframing, and make reframings visible as such.
  • Bring in the expert for the thing you can't check. The lab hired the vault's co-discoverer. The wiki's #to-read leads are the thin version of this.
  • Feed the errors back. The lab's notes went to the people training Claude. A wiki-scale analogue is logging where review found something, so the pattern is known, rather than only logging what was written.

A structural counterweight (auto-ingest 2026-11-08, third source). Anthropic — Progress on Provable Inference for Large Language Models faces T3's problem in engineering form. Claude built a cryptographic system whose design choices are "hard to evaluate." Anthropic did not ask reviewers to understand every choice. They made the claim small and mechanically checkable (Lean statements plus a trusted verifier) so that "you can fully distrust the prover if you trust the verifier." The human then audits only the statements, which is "dramatically less onerous than auditing proofs" (Provable Inference).

Read onto this wiki: review gets cheaper when the agent's claims are shaped for checking, not when the reviewer knows more. Candidate (the human's call, not adopted): every ## Key claims bullet carries a verbatim quote or figure from raw/, so a PR review is a spot-check of quote against source, not a re-derivation of the argument. This overlaps with T5's verbatim-over-summary finding. Its limit is the same as FlashProve's: someone still has to judge whether the claim is the right one, and that part stays knowledge-bound. Syntheses and connections, the wiki's most valuable output, are also the least formalizable.

T4 — Automation vs. engagement open · opened 2026-11-08

The wiki now auto-ingests subscribed feeds (Anthropic Research first) via a daily job that opens a PR and alerts on Slack (see CLAUDE.md, Subscribed feeds). That buys coverage and freshness. The cost is that it removes the INGEST "discuss before writing" step, one of T3's two counterweights. Karpathy's own preference was the other branch: "I prefer to ingest sources one at a time and stay involved," with batch ingest "with less supervision" offered as the alternative (Karpathy — LLM Wiki Gist).

Mitigation adopted: the discussion is moved, not deleted. It becomes the PR summary (takeaways, "Needs your call," questions), and pages stay tagged auto-ingested until reviewed. Seeds are never promoted unattended.

How we'll know it's failing: the review queue below keeps growing, or PRs get merged without being read. The planned front door makes the queue a daily surface: Front Door Pattern Language pattern 6 draws its first todo items from auto-ingested pages, oldest first, and counts "done" only when the tag is gone from the wiki. Then the wiki is accumulating text the human never thought with, which is exactly Willingham's warning in T3.

Sharpened (auto-ingest 2026-11-08). Anthropic — Teaching AI to Build Scientific Models states T4's worry as a claim about science: "A scientific model may provide an answer, but it is the process of constructing it that provides the insight." Hodgkin and Huxley "understood what the model was telling them because they had built it themselves, and they knew what hadn't worked." Read onto this wiki: an unattended ingest gives the human the product (pages) without the process (reading the source, trying framings, rejecting links). The current mitigation (the PR summary) moves the takeaways, but not the dead ends.

The source's remedy is specific: when AI does the building, it must hand over "which parts are pinned down by the data, which are still open, which alternatives failed." Candidate (the human's call, not adopted): add a short "Considered and rejected" section to the auto-ingest summary. It would list links the agent judged too weak, pages it chose not to create, and verdicts it found borderline (e.g. contradiction vs scope note). Then review covers the agent's eliminations as well as its output, which also gives T3's reviewer a cheap check (Distributions over Best Fits). The summary for this ingest includes a trial version.

T5 — The wiki as agent memory: evidence on the method itself partly resolved · opened 2026-11-08 (auto-ingest)

Verdict (human, 2026-11-08): adopted three responses.

  1. CLAUDE.md now frames the wiki as the map on top of raw/ + search (the territory). QUERY searches both.
  2. Every ## Key claims bullet quotes raw/ verbatim or gives the exact figure.
  3. The "10–15 pages per source" target is dropped: touch only the pages a source changes.

Plus a Borges check: scripts/lint/borges.py runs in LINT (The Borges Map). Its first run found two Borges clusters: engineering craft at 11.1× its source and Wiener at 9.8×. Compressing them is the next step, tied to T1's worked examples. Still open: long-instruction adherence, dream cadence, and multi-writer merges. Anthropic — The Components of Agent Memory evaluates the Agent-Maintained Wiki pattern this wiki is built on. It is the first source in the wiki that tests the method directly instead of by analogy. The evidence cuts both ways.

Affirms:

  • Compounding: 13% → 48% as dreaming scaled from 2 to 128 sessions.
  • Linking pays on synthesis: 87% against 73% on cross-document questions. This supports T2.
  • Matching infrastructure: log.md works like the shared coordination log, index.md like the injected index, and LINT like the gap-filling dream pass.
  • Provenance states: marking claims as checked or unconfirmed is what the strongest models learned to do in their own notes.

Contradiction (resolved 2026-11-08 by rewording CLAUDE.md; kept for history): CLAUDE.md said "you are not a RAG system that rediscovers knowledge on every query" and treats the wiki as the replacement for re-reading sources. Anthropic — The Components of Agent Memory finds wiki-only (48%) scored below search-only (53%). Wiki + search (61%) beat both. The wiki added nothing on single-document lookups. Read this way, the wiki is a map on top of search, not a substitute for it. (qmd over both wiki and raw already allows that in practice; the manual's framing doesn't say so.)

Challenges (candidate responses, the human's call):

  • Summary loses facts. Adopted: verbatim quotes in Key claims.
  • Borges bloat. Adopted: scripts/lint/borges.py; see The Borges Map.
  • Salience over hoarding. Adopted: page quota dropped; store what's worked out.
  • Long instructions reduce adherence. The source warns of this directly. CLAUDE.md is long and growing. Candidate: keep a short top-layer manual and push operation details into skill files, which this repo partly does already.
  • Dream cadence. LINT runs only when asked. With auto-ingest adding pages daily (T4), consider a scheduled LINT. The source suggests focusing it on volatile pages.
  • Many writers, one page. Agents writing one shared page overwrote each other. Parallel auto-ingest branches all edit index.md and log.md, which will show up as merge conflicts. Merge order is a human review point.

Tension on what to keep (auto-ingest 2026-11-08). Salience says keep little. Anthropic — Teaching AI to Build Scientific Models says the opposite about models: "we keep both likely and unlikely models, as all models can carry insight," and an interpretation is only as strong as the alternatives it ruled out. Recorded as a tension, not a contradiction, because the two reconcile under Agent Memory's own rule, "persist what was expensive to work out": a ruled-out alternative is expensive to work out, so it counts as salient. The wiki already half-follows this, since PROMOTE keeps refuted seeds as "documented dead-ends." The open part: INGEST records what a source says but not which readings or links were tried and dropped. See the T4 candidate above.

Sharpened 2026-11-08 (human-requested ingest). Blumenfeld — Scaling Karpathy's LLM Wiki (Neo4j) names a scaling dimension T5 had not: the cost of reading and navigating the wiki, which on a file system grows with its size ("a k-hop traversal costs you k scans … what points at this page — has no index behind it at all"). The quality consequence lands on INGEST step 3: an agent that cannot look around cheaply before writing produces duplicate pages and missed links, "and now you're persisting it." The evidence is vendor-adjacent and does not test the wiki pattern itself (see the source page's caveats), but the mechanism is sound and this wiki has already felt it: the auto-ingest run lost qmd and fell back to grep for a week of ingests.

Verdict (2026-11-08, with the human): adopt the data model, defer the database. The front door's graph.json already is Blumenfeld's model (documents, sections, links, backlinks, aliases, deterministic rebuild, no LLM). It now has a second reader: a small CLI for the agent (site/scripts/graph-query.mjs), so INGEST and LINT can ask for backlinks, neighbours, orphans and unresolved links by pointer instead of by scan (Navigation vs Search). Neo4j itself is deferred until a scan is actually felt. Considered and rejected: a new tension (this is T5's scaling branch, not a new argument); an entity page for Neo4j or the author (lean). Done the same day: section-level retrieval. The build records heading-bounded sections with line ranges, backlinks deep-link to the section they come from, and graph-query.mjs section "Method Reflection" t4 returns one tension instead of this whole page; this bears on "long instructions reduce adherence" above.

Bearing on T3. The source is about memory for the agent. T3 asks whether the human ends up holding the knowledge. A wiki tuned to be the best agent memory could be the "substitute" branch of T3 by design.

What has held up (method decisions, with dates)

  • Three provenance states (2026-06-20): added draft/#unsourced so established facts aren't mislabelled as hunches.
  • Auto-grounding (2026-06-20): established-fact captures get a canonical source the same day.
  • Inline sources persisted to raw/ (2026-06-26): curation-by-proxy keeps every source file durable.
  • Refuted seeds kept as dead-ends, not deleted (PROMOTE rule): affirmed 2026-11-08 by Anthropic — Teaching AI to Build Scientific Models, where an interpretation's weight comes from the alternatives it ruled out.
  • Seeds evaluated, not force-promoted: both seeds have twice been assessed and correctly left unpromoted, so the gate is working.

Live views (Dataview)

Pages that carry a method tension:

LIST FROM "wiki" WHERE contains(tags, "method-tension")

Maxim pages and how many sources back each (1 = the human's list alone; T1 targets):

TABLE source-count AS sources, updated
FROM "wiki/concepts"
WHERE contains(tags, "craft") OR contains(tags, "heuristics")
SORT source-count ASC

Auto-ingested pages awaiting your review (T4; remove the tag once read):

TABLE file.folder AS folder, created
FROM "wiki" WHERE contains(tags, "auto-ingested")
SORT created ASC

Unverified hunches and reading leads (the promise-to-verify backlog):

LIST FROM "wiki" WHERE status = "seed" OR contains(tags, "to-read")

Connections

The wiki is the source. This door reads it at build time and never writes to it.