On a chatbot running on a local council website, a very simple question produced a wrong answer: who looks after the finances? The bot replied with a name, a role, and the present tense.
The name was real. The role was correct, but it was correct in 2019: the person had left the executive at the following term. The answer came from a PDF of meeting minutes, indexed like everything else on the site.
The bot hallucinated nothing. It presented an out of date fact as today’s situation, with the same confidence it uses for everything else. That is how stale data in RAG usually gets out: not as an invention, but as a real fact that has expired.
Why stale data in RAG slips through unnoticed
Two reasons stack up: the archive wins on score, and nothing flags that it describes the past.
Minutes of a meeting contain full sentences: Mr X, councillor in charge of finance, presents the budget. The official page listing the executive looks more like a table of names with job titles next to them.
So for who is in charge of finance?, the 2019 PDF is a better match than the correct page: it is chattier and closer to how the question is phrased.
The real problem sits one step later. Once the chunk has been retrieved, nothing inside it says that it describes the past.
The model receives a paragraph in the present tense, with no date. It has no way of knowing that this present tense is five years old, so it copies it.
Forcing the current page into the context
First fix: the official page enters the context without going through the score, and the archives are dropped.
if (this.isAuthorityContext(ctx)) {
filters.pdf = false;
filters.linked = ['https://example.ch/executive'];
filters.linkedAlwaysInclude = true;
}That block is useless if it never fires. At first it depended on routing: the question had to be classified as being about people or about authorities.
In practice, routing missed about half of the cases. Who looks after tourism? contains no word that looks like an authority.
So I stopped trusting the routing and detected these questions by vocabulary instead. The site is French speaking, hence the French role words.
// the role words, with a guard on each side so a longer word that contains
// one of them does not match
const ROLES = /(?<!\p{L})(?:syndic|syndique|municipalité|municipa(?:l|ux)|dicastères?)(?!\p{L})/iu;
// a responsibility phrasing, then a policy area, within 40 characters
const AREAS = /(?:responsable|chargée?|s'occupe|en charge)[^.?!]{0,40}(?:finances|tourisme|voirie|sports|culture)/iu;
private isAuthorityContext(ctx: RequestContext): boolean {
if (ctx.hasRoute('people') || ctx.hasRoute('government')) {
return true;
}
const question = ctx.getCleanedQuestion() ?? '';
return ROLES.test(question) || AREAS.test(question);
}Then a second gap showed up, a nastier one: the question that is only a name. Firstname Lastname, no verb and no role.
There is no role vocabulary to detect, so the rule never fires and the archives move back to the front. That needed its own detection.
Dating the archives so the model speaks in the past tense
Excluding the archives fixes one case and breaks another. Better to keep them and make their age readable.
Dropping the archives does remove the stale data in RAG answers about the current composition. But if somebody asks about a person who has left, the archive is the only place on the site where they appear, and the bot then answers that it found nothing.
The fix happens at indexing time rather than in the prompt. Every chunk of an archive PDF gets its date and a warning prepended to it.
private markArchiveDocumentAsDated(document: ZaasDocument): void {
if (document.metadata.type !== 'pdf') {
return;
}
const date = this.getArchiveDateLabel(document);
document.pageContent =
`[Archive document${date ? ` dated ${date}` : ''}, describes a past ` +
`situation, do not present its content as the current situation]\n` +
document.pageContent;
}The date is looked up in a cascade: the document metadata, then a date in the URL, then just the year. A year is enough to tip the model into the past tense.
The system prompt then sets the ranking of the sources, leaning on that marker:
- minutes, proposals and management reports describe a past situation and never prove that a person currently holds a role;
- archive extracts are introduced by
[Archive document dated ...], so use the past tense for everything they describe; - the only authoritative source for the current composition is the up to date official page of the executive;
- if the person asked about does not appear on that page, say so explicitly, then give, in the past tense, the role they held according to the archives.
That last rule is the one that changes the experience. The bot no longer merely avoids being wrong, it explains: this person was the councillor in charge of that area, they no longer are, and here is the current executive.
The little problem: the wrong gendered title
One last detail took me three attempts, and it carries the most reusable lesson.
French job titles are gendered, so a council leader is le syndic if he is a man and la syndique if she is a woman. Asked who is la syndique?, the bot answered La syndique is Mr X. It echoed the gender in the question instead of the gender of the person.
My first rule said make the title agree with the real gender of the person. It changed nothing, for a good reason: the model cannot know somebody’s gender from their name. I was asking it to guess.
The second said copy the title exactly as it appears on the official page. Better, but the bot still drifted whenever the question insisted.
The third one worked, because it points at a concrete marker sitting in the retrieved text:
Ignore the form used in the question. Work out the gender of the person in office from the up to date official page (
M.= man,Mme= woman), then use le syndic for a man and la syndique for a woman.
A prompt rule that asks the model to work something out fails. A rule that shows it where to look in the context holds.
The general shape of the fix comes down to two layers: retrieval has to guarantee that the current source is present without depending on a score, and indexing has to make the age of a document readable in the text. Neither the prompt on its own nor the vector search on its own gets you there, and that is why stale data in RAG keeps surprising people who only tune one of the two.