The short version
An instruction such as "Only cite real cases" or "Never fabricate citations" is added to a prompt, and the output is then treated as if it had been checked. The instruction can change how the model words its answer. It cannot look anything up.
What it is
The belief that a fact about the world can be fixed by asking for it. The instruction sits in a research prompt, in a tool's custom instructions or in a firm's shared prompt library, and over time it starts to stand in for the check it was meant to accompany.
The guide on this site that carries an instruction of this kind, The Citation Verification Rule, says of it: "It is not a check." The courts say the same thing from the other side. The duty set out in R (Ayinde) v London Borough of Haringey and Al-Haroun v Qatar National Bank [2025] EWHC 1383 (Admin), judgment of 6 June 2025, is to check the accuracy of AI research "by reference to authoritative sources" (paragraph 7). A model's compliance with an instruction is not an authoritative source. Read the judgment at caselaw.nationalarchives.gov.uk/ewhc/admin/2025/1383.
A sighting
The example below is made up to show the pattern. A trainee's research prompt ends: "Only cite real cases. Do not make anything up." The answer lists four authorities, each with a neutral citation and a two-line summary, and none of them carries a flag. The trainee's note to the supervising solicitor says the citations were "checked by prompt". Nobody has opened any of them.
The answer looks the same whether the cases exist or not, and that is the trouble. The instruction has added nothing a reader can check: there is still no source in the answer, only a model's assurance.
Why it happens
Prompting does change output, so the habit spreads from cases where it works. "Answer in British English" works because spelling is something the model controls. Whether a case exists is not: it is a fact in a law report that the model cannot see, unless the tool searches a source and shows which one.
The judicial guidance puts it this way: "Public AI chatbots do not provide answers from authoritative databases. They generate new text using an algorithm based on the prompts they receive and the data they have been trained upon." It adds: "Even with the best prompts, the information provided may be inaccurate, incomplete, misleading, or biased" (Artificial Intelligence (AI): Guidance for Judicial Office Holders, 31 October 2025, section 3, part I).
The Bar Council's guidance describes what was going on when a lawyer questioned a chatbot about its cases: "all the LLM was doing was producing outputs (which just happened to be in the form of words) which its mathematical processes related to its inputs" (Considerations when using ChatGPT and generative artificial intelligence software based on large language models, last reviewed 25 November 2025, paragraph 19). An instruction is one more input.
What to say back
"That line tells the model what to say about its citations; it does not check them. Which of these has someone opened on the National Archives, BAILII or a subscription service?"
Origin
- 22 June 2023. In Mata v Avianca, a United States case, the court recorded what a chatbot did with requests for authority: "When directed to 'provide case law', 'show me specific holdings', 'show me more cases' and 'give me some cases', the chatbot complied by making them up" (Opinion and Order on Sanctions, Findings of Fact, paragraph 39). An instruction to cite only real cases is more words of the same kind.
- 30 January 2024. The Bar Council issues its guidance on ChatGPT and generative AI for barristers. The version last reviewed on 25 November 2025 explains why a chatbot's answers about its own sources are outputs, not checks (paragraph 19).
- 6 June 2025. Ayinde and Al-Haroun: the duty to check against authoritative sources (paragraph 7).
- 31 October 2025. The refreshed judicial guidance: "Even with the best prompts, the information provided may be inaccurate".
- Now. The Citation Verification Rule on this site still recommends an instruction of this kind, because it asks the AI to flag uncertainty instead of inventing a citation, and still says that every citation has to be verified, flagged or not. The first use of the phrase is not recorded.