← All posts
Knowledge ManagementNovember 5, 2026

Karpathy's LLM wiki — with a citation on every line

An LLM wiki is only as trustworthy as its check against the sources. What went wrong when we built Pith's pages on EU regulation, what we changed, and the wrong dates the citation check caught before they were stored.

In short

An LLM wiki is a set of pages a language model writes and keeps up to date from your sources, so knowledge is compiled once instead of re-derived for every question. It is only as trustworthy as the check that compares each sentence with the source it cites. In Pith, every paragraph cites a saved source, and a page is stored only if the citation check finds no unsupported or wrongly cited statement; pages that are published also need a human approval.

S

Stefan Hartmann

Builds Pith

Karpathy's LLM wiki — with a citation on every line

An LLM wiki is a set of pages a language model writes and keeps up to date from your sources, so knowledge is compiled once instead of re-derived for every question. It is only as trustworthy as the check that holds each sentence against the source it cites. We learned that while building pages on EU regulation in Pith in October and November 2026: the model wrote fluent, plausible pages from day one, and the check caught dates that were wrong in a very believable way.

This post is the honest version of that build: what broke, what we changed, and what still needs a person.

What is an LLM wiki?

Andrej Karpathy published a short idea file describing a different way to use language models with documents. Instead of retrieving chunks from raw files for every question, the model reads each new source once and works it into a set of interlinked pages: it updates existing pages, adds new ones and notes where a new source contradicts an old claim. His summary: "the wiki is a persistent, compounding artifact."

We have written a longer guide to the LLM wiki, a comparison with RAG and a glossary entry. The short version: RAG answers and forgets; an LLM wiki writes things down and keeps them.

Pith builds an LLM wiki from what you save. The one thing we add to the pattern, strictly, is evidence: every paragraph cites the saved source it rests on, and a check compares the claim with that source before a page is stored. How that holds up on hard material is the rest of this post.

Why is EU regulation a hard test?

We are building public pages on EU acts that come up again and again in compliance work: NIS2, DORA, the Cyber Resilience Act, eIDAS and others. They are not public yet; each one waits for review.

Legal texts are a good stress test for an LLM wiki. They are long, the sentences that matter are buried in articles and paragraphs, and many dates sit close together: adoption, publication, entry into force, application. A page that mixes them up still reads perfectly well.

What went wrong in the first build?

The first version of these pages looked finished and was not. Three problems:

  • The pages explained why, not what. The wiki agent did not see the full act. It saw a summary of the first 8,000 characters of each act, and in an EU act those are the recitals: the reasons the law was made. So the pages explained the motivation well and said little about who is covered, what the obligations are or when they apply — exactly the parts a reader comes for.
  • Two languages on one page. Some sources are in English. In a German workspace, they produced English paragraphs inside German pages.
  • A citation that pointed nowhere. One citation marker referred to no source at all.

None of this was a hallucination in the usual sense. Every sentence rested on something. The model had been given the wrong part of the text and rules that were too loose.

How did we fix what the model reads?

We changed the inputs before we touched the prompts:

  • Articles, not recitals. EU acts are now summarised from their enacting terms, with the article numbers kept, so a claim can point to a specific article instead of "the directive".
  • One language per wiki. The wiki of a workspace is written in one language. An English source feeds a German page in German.
  • No dead markers. A citation marker that points at no source is dropped instead of shown.

What does the "restructure page" step do?

A wiki page grows source by source, and after ten sources it reads like ten additions. For regulation pages we added a step that rewrites a grown page into a fixed outline: overview, who is covered, obligations, reporting, deadlines, supervision and penalties, and implementation in Germany.

On the way, the step:

  • merges duplicate statements that arrived from different sources,
  • re-attaches each citation to the source that actually states the fact, not one that merely mentions the topic,
  • fills thin sections from passages of the sources the page already cites, and nothing from outside,
  • and saves the result only if the citation check finds no unsupported or wrongly cited statement. Otherwise the previous version stays.

The last rule is the one that matters. A restructure that makes a page tidier but less true does not get stored.

What did the citation check catch?

Two examples from the real run.

A believable wrong date. For NIS2 and DORA, a model proposed "in force since 27 December 2022". The date is real: it is the day both acts were published in the Official Journal. But both entered into force on 16 January 2023. The check compared the claim with the cited text, flagged it as a wrong citation, and the statement was not stored. Someone skimming the page would very likely have accepted it.

An imprecise paraphrase. On the Cyber Resilience Act page, a sentence said security updates must be "provided for at least ten years". That is not what the text says. Art. 13(9) requires that each security update remains available for at least ten years after it has been made available. For a manufacturer planning a support period that is a real difference, and the sentence was corrected.

This is what we think a citation check is for. Not catching nonsense, which is easy to spot, but catching sentences that are almost right.

What did the model still miss?

A check can only judge what is on the page. It cannot tell you what should be there and is not. In a hand review of the CRA page we found that Art. 69(3) was missing: the reporting obligations also apply to products placed on the market before 11 December 2027. No model step had flagged it, because no sentence was wrong. One was absent.

So human review stays part of the loop. For the public regulation pages the rule is simple: a page goes public only after we have read it and approved a specific version.

Why did long legal texts break retrieval?

Filling a thin section means finding the right passage in a long source. On legal texts that failed in a way we had not expected. In the eIDAS amending act, one article sat in a single huge paragraph; amending acts insert whole new articles into the regulation they change, and the inserted text came through as one block. A search for one obligation returned a passage about everything.

The fix had three parts: split long paragraphs, split inserted articles into chunks of their own, and prefix every chunk with its article heading. A chunk now knows which article it belongs to, even when it is the fourth piece of it.

What does this cost?

Less than we expected. Filling cited fields for 21 pages cost about €0.06 in model calls. Re-chunking five EU acts cost about €0.014. The expensive part is not the model; it is the time a person spends on review. That is why we want the machine checks to catch as much as possible before someone reads.

What do we take from it?

An LLM wiki is only as trustworthy as its check against the sources. Fluent pages are cheap; the model produced good-looking ones from the first day. What made them usable was one rule — a sentence is stored only if the passage it cites says the same thing — and a person who reads the page before it goes out.

If you keep your own LLM wiki, in Pith or elsewhere, the lessons carry over:

  • give the model the part of a source that contains the facts, not the part that happens to come first,
  • keep each citation attached to the passage that states the fact,
  • check every change against the sources before it replaces the previous version,
  • and keep a human approval for anything you publish.

How the wiki works in Pith is on the wiki feature page; why we start from citations is in Every claim cites its source. The curated EU dates are already on our public deadline timeline, each linked to the passage that states it.

FAQ

What is an LLM wiki?

A set of interlinked pages that a language model writes from the sources you collect and extends as new sources arrive. The idea was described by Andrej Karpathy in a short idea file on GitHub. Unlike retrieval at question time, the synthesis is written down once and kept.

What does a citation check catch that a reader would miss?

Sentences that are almost right. In our build, a model proposed that NIS2 and DORA were 'in force since 27 December 2022'. That is the date both acts were published in the Official Journal; both entered into force on 16 January 2023. The check compared the claim with the cited text, flagged it as wrongly cited, and it was not stored.

Does a citation check make human review unnecessary?

No. A check can judge what is on a page, not what is missing from it. In a hand review of our Cyber Resilience Act page we found that Art. 69(3) was missing: reporting obligations also apply to products placed on the market before 11 December 2027. Pages we publish therefore go public only after we approve a specific version.

Why do LLM wikis struggle with legal texts?

Two reasons we ran into: the first part of an EU act is the recitals, so a model that reads only the beginning learns why a law was made but not what it requires; and amending acts can put a whole inserted article into one huge paragraph, which breaks retrieval. We now summarise from the articles and split long paragraphs, with each chunk prefixed by its article heading.

What does running these checks cost?

Little in model calls. Filling cited fields for 21 pages cost about €0.06, and re-chunking five EU acts about €0.014. The larger cost is the time a person spends on review.

Related