The short version
A question goes into a chatbot as it would into a search engine, and the answer is read as if something had been found. A public chatbot of the kind the judicial guidance describes does not look the answer up in an authoritative database. It writes the words most likely to come next, and a case name or a link can be among them.
What it is
A search engine returns documents that other people wrote, each at an address that can be opened. A chatbot returns text it has just composed. On screen the two look alike: a box, a question, an answer.
The judicial guidance spells out the difference. Public AI chatbots "generate new text using an algorithm based on the prompts they receive and the data they have been trained upon", so their output "is what the model predicts to be the most likely combination of words", which "is not necessarily the most accurate answer". Such tools, it says, "may be useful to find material you would recognise as correct but have not got to hand, but are a poor way of conducting research to find new information you cannot verify" (Artificial Intelligence (AI): Guidance for Judicial Office Holders, 31 October 2025, section 3, part I).
The SRA said the same in its Risk Outlook report on AI of 20 November 2023: language models such as ChatGPT "work by anticipating the text that should follow the input they are given, but do not have a concept of 'reality'" (SRA). The Divisional Court quoted that passage in Ayinde (paragraph 13).
The line has also blurred from the search side. Google's help page on its AI Overviews, checked on 1 October 2026, says that they "use generative AI" and that "AI Overviews can and will make mistakes" (Google Search Help). A results page can now open with text that a model wrote.
A sighting
The earliest record cited here is from the United States. In Mata v Avianca, one of the plaintiff's lawyers explained why he had turned to ChatGPT: "I heard about this new site which I assumed -- I falsely assumed was like a super search engine called ChatGPT, and that's what I used" (Opinion and Order on Sanctions, 22 June 2023, Findings of Fact, paragraph 37).
In England and Wales an account of it appears in the judgment in R (Ayinde) v London Borough of Haringey and Al-Haroun v Qatar National Bank [2025] EWHC 1383 (Admin), of 6 June 2025. Counsel in Ayinde denied using AI tools for legal research (paragraph 53). The court records that, in a second witness statement, counsel said that when drafting the grounds she "may also have carried out searches on Google or Safari"; in the court's words, she may have taken account of "artificial intelligence generated summaries of the results (without realising what they were)" (paragraph 57). The court had been given no evidence for a contention, "which in any event, was not directly advanced", that the fake cases had emerged that way (paragraph 66), and said that counsel had not provided "a coherent explanation for what happened" (paragraph 67).
In Al-Haroun, the court summarised the client's account: the citations "were generated using publicly available artificial intelligence tools, legal search engines and online sources" (paragraph 76). The paragraph names three kinds of source and does not say which produced which citation. Read the judgment at caselaw.nationalarchives.gov.uk/ewhc/admin/2025/1383.
Why it happens
The habit was formed on search engines, where a result points to a document and the reader decides whether to trust the document. Behind a chatbot's answer there may be no document at all. In Mata, the lawyer "did not have the full text of any 'decision' generated by ChatGPT", and "cited and quoted only from excerpts generated by the chatbot" (Findings of Fact, paragraph 40).
The Divisional Court did not treat a genuine search result as safe either. The duty to check, it said, is "no different from the responsibility of a lawyer who relies on the work of a trainee solicitor or a pupil barrister for example, or on information obtained from an internet search" (paragraph 8). The check is against authoritative sources, which the court listed, from "the Government's database of legislation" to "the databases of reputable legal publishers" (paragraph 7).
What to say back
"Did the tool find this, or write it? Which document is it from, and has anyone opened that document on the National Archives, legislation.gov.uk or a publisher's database?"
Origin
- 22 June 2023. In Mata v Avianca, a United States case, the opinion on sanctions records a lawyer's evidence that he had taken ChatGPT for "a super search engine" (Findings of Fact, paragraph 37).
- 20 November 2023. The SRA's Risk Outlook report on AI: language models predict text and have no concept of reality.
- 6 June 2025. Ayinde and Al-Haroun: the duty to check applies to information from an internet search as it does to AI research (paragraph 8), and counsel's account of searches on Google or Safari is recorded (paragraph 57).
- 31 October 2025. The current judicial guidance, quoted above, replaces the version of April 2025.
- Now. On 1 October 2026, Google's help page said that AI Overviews "can and will make mistakes" and told readers: "Always check important info in more than one place."