Our LLM wiki invented six articles. Here's the check that stops it.
A news site sent Pith its cookie notice instead of the article. Pith summarised what it guessed the article said, and the wiki built six pages on those guesses. What went wrong, the readability gate we built, the numbers from testing it, and why "I couldn't read this" is the right answer.
In short
When a web fetch returns a cookie notice, paywall or bot check instead of the article, a language model asked to summarise it may fill the gap from the title and URL and produce a plausible but invented summary. Pith now runs a readability gate before anything is derived from a fetched text: one model call labels the text as article, consent or paywall, login or bot wall, error, or other non-content, and only text with a probability of at least 0.3 for 'article' is summarised, indexed for search or used in the wiki. Anything else stays a link with the reason shown.
Builds Pith

On 6 November I saved a batch of articles from my own reading into my Personal workspace in Pith. Nine of them were from golem.de. Shortly afterwards my wiki had six new pages: on redacted PDFs at public authorities, on neurotechnology, on the Tesla Roadster, on Musk's election donations. Each page was fluent, sensibly structured and cited one source.
None of it came from the articles. Pith had never read them.
This post is about what happened, what I built so it does not happen again, and the numbers from testing it. It is also a correction of something I had assumed about my own product.
What went wrong?
When Pith saves a link, it fetches the page, stores the text, summarises it, splits it into chunks for search, and hands it to the wiki agent, which decides whether it belongs on a page. That chain assumed one thing it never checked: that the fetched text was the article.
For all nine Golem links, it was not. Golem answered the fetch with its cookie-consent and "Pur-Abo" notice: about 2,500 characters that start with "Cookies zustimmen" and say nothing about the article. To the fetch, this looked like a success. The server answered, there was a body and a title.
Then the summariser got that text, plus the title and the URL. Three of the nine summaries said honestly that the page only showed a cookie notice. Six did not. They described the article the title and URL slug suggested, plausibly and with no basis in the stored text. And the wiki agent, seeing six new sources on topics the wiki did not cover yet, did its job and created six new pages, each citing only one of those sources.
So the wiki said "source: golem.de" under paragraphs that golem.de never sent us.
Why does a model fill the gap?
Because that is what it was asked to do. A summariser is given a title, a URL and a text, with the instruction to summarise the article. If the text is a cookie notice, the honest answer is "there is no article here". But a news headline plus a descriptive URL slug is a lot of material. "Tesla Roadster" and a date are enough for a model to write a believable paragraph about the Tesla Roadster. Nothing in the output looks wrong, and that is the problem.
I write a lot about source grounding here: every claim in the LLM wiki should lead to something you saved. The citation check compares sentences with the cited source. But a check like that can only work on what is stored. If the stored "source" is a guess, the wiki is grounded in the guess. This was an AI hallucination one step earlier than where I was looking.
How big was it?
I cleaned up the same day: deleted the six pages, removed the summaries, search chunks and stored text of the nine Golem sources, and kept them as links with their real titles. Then I went looking for the same pattern elsewhere.
It was there. In one of the public showcase workspaces, ten Finextra pages had come back as a bot wall of 275 characters. Those had produced claims in four wiki pages, plus one page built only on them. I removed or rewrote them. I also found a heise+ "Leseprobe" (a preview of a subscriber article) and a Harvard Business Review teaser that had been treated as full articles.
None of these came from an attacker or an unusual site. They are ordinary news pages behaving as they do for any visitor who has not clicked "accept" or logged in.
What does the readability gate do?
Before anything is derived from a fetched text, Pith now asks one question: is this the page's content, or something standing in front of it?
It does this with one call to OpenAI's Decisions API, using the gpt-6-luna model. The text is labelled as one of five things:
- article: the actual content, even with menus and cookie banners around it
- consent_or_paywall: a cookie notice, subscription offer or paywall instead of the content
- login_or_bot_wall: a login prompt, bot check or access-denied page
- error_or_empty: an error page or a page with no real text
- other_non_content: only navigation, a list of links, a redirect notice
The text counts as readable if the probability for "article" is at least 0.3. Text with fewer than 200 non-blank characters is marked too short without a model call at all.
If a source is not readable, there is no summary, no search chunks and no wiki. The source stays in your list as a link with its title, and Pith tells you why, for example: "Pith couldn't read this page: it showed a cookie-consent or paywall notice instead of the article."
The check costs about €0.0002 per fetched page and takes roughly 0.2 to 1 second, in the background after the save. It is a system cost; you are not billed for it. If the call fails or times out, the save still goes through, and the simple length check decides.
Why did the first version fail on real articles?
Because it looked at the wrong part of the page.
A news page is long. To keep the call small, the first version showed the model the first 5,000 and the last 1,000 characters of the text. On many sites that is the menu and the footer. The article sits in between. So long, real articles from The Register, the ECB and the FSB were judged as non-content, at a probability of around 0.35 for "article".
The fix is structural, not a list of words. Pith now finds the page's prose: long lines that are not mostly links. Consecutive prose lines form a run. An article body is one long run; teaser cards, login promos and banners are short ones. The model sees the head of the page, then the longest prose runs in full and the start of the others, in page order, then the tail.
I also tried a different question along the way: a plain yes/no predicate, "does this text contain body text?". It was worse. It gave 0.0 for real articles from the FSB, ComputerBase and AWS. I dropped it.
How well does it work?
I ran the final version against production on 6 November with 25 cases: 9 prepared fixtures and 16 real sources, walls and articles mixed. 24 of 25 came out as expected.
| Page | Probability of "article" |
|---|---|
| Real articles (range) | 0.94–1.00 |
| InfoQ article with login promos | 0.99 (an earlier version: 0.06) |
| The New Stack article | 0.98 (an earlier version: 0.39) |
| Golem cookie / Pur-Abo notice | 0.00 |
| Finextra bot wall | 0.00 |
| heise+ Leseprobe | 0.00–0.02 |
| Wikipedia "not found" page | 0.02 |
| HBR teaser | 0.34 (the miss) |
The miss is the HBR teaser. At 0.34 it is just above the line, so the gate let it through as an article. That is why there is a second guard.
What is the second guard?
The summariser has a new rule: use only the text it was given, never reconstruct an article from its title or URL, and if the text is not the article, answer with exactly NO_CONTENT and nothing else. When it does, the source is marked unreadable the same way, and its text and chunks are removed. Borderline cases like the HBR teaser are what it is for — though that particular one I cleaned up by hand before the rule existed.
The wiki agent also refuses sources that are marked unreadable. So a page that gets past the gate still has to get past the summariser, and a source marked unreadable never reaches a wiki page.
Why treat paywall previews as unreadable?
This was a deliberate choice. A paywall preview shows a few paragraphs, then "subscribe to read more". Those paragraphs are real text from the article. But a summary of a teaser reads exactly like a summary of the whole article, and you cannot tell from the summary which one you got. An introduction often promises what the article then qualifies. So a teaser counts as unreadable, the same as a wall.
What if I can read the article and Pith can't?
That is common. You accepted the cookies; you have the subscription. Pith's fetch has done neither.
Now the Chrome extension can send the text you are reading along with the save. If Pith could not read a page you saved, the extension offers to add the text you see. That text goes through the same readability gate as a fetch, so a captured cookie banner is still not an article. It is never stored in public workspaces; their sources come only from the normal fetch. And publishers that reserve their content against text and data mining stay link-only, whatever the extension sends.
What did I learn?
That a source-grounded wiki has to be able to say "I couldn't read this".
I had built several checks that compare what the wiki says with what the sources say. They all assumed the source was real. The weakest point was before any of them: the moment a cookie notice was stored as an article and a model was asked, politely, to summarise it.
A plausible invention is worse than a gap. A gap you notice. A fluent page with a citation you trust, copy into a client note and repeat. Showing "Pith couldn't read this page" is less impressive than a summary. It is also true.
The gate does not make the wiki perfect. A teaser can still land just above the line, and the second guard has to catch it. But the failure I saw on 6 November, pages built on articles nobody read, now has to get past two checks instead of none.
More on how Pith keeps pages tied to their sources: the wiki, the guide to the LLM wiki, and Karpathy's LLM wiki — with a citation on every line.
FAQ
Why would a model invent an article it never saw?
Because it was asked to summarise and was given a title and a URL alongside a text that was not the article. A title like a news headline plus a descriptive URL slug is enough for a fluent, plausible summary. The model does what it was asked; the mistake is handing it a cookie notice and calling it the article.
What does Pith do with a page it cannot read?
It keeps the source as a link with its title and shows why: for example 'Pith couldn't read this page: it showed a cookie-consent or paywall notice instead of the article.' There is no summary, the text is not indexed for search, and the wiki does not use it.
Are paywall previews treated as readable?
No, on purpose. A preview of a few paragraphs followed by a subscription offer would produce a summary that reads like a summary of the whole article. Pith treats such teasers as unreadable.
How do I get a walled article into Pith?
With the Chrome extension. After you have accepted the cookies, or when you are subscribed, the extension can send the text you are reading with the save. That text goes through the same readability check. It is never stored in public workspaces, and publishers that reserve their content against text and data mining stay link-only.
What does the check cost?
About €0.0002 per fetched page, as a system cost that is not billed to you. It runs in the background after a save and takes roughly 0.2 to 1 second.