The short version
"Hallucination-free" citations, hallucinations "eliminated" or "avoided": the vendors' descriptions of their own research tools. When researchers at Stanford measured three such tools in 2024, each still gave false or misgrounded answers to between a sixth and a third of the questions put to it.
What it is
A claim that covers a narrow thing and is heard as a wide one. The narrow thing may be that every linked citation opens a real document. The wide thing a buyer hears is that the answer is right. Those are different promises. A real case, correctly linked, can still be cited for a point it does not decide.
The Stanford study, by Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D. Manning and Daniel E. Ho, set out from the claims themselves. Its abstract records that providers had described retrieval-augmented generation as "eliminating" (Casetext) or "avoid[ing]" (Thomson Reuters) hallucinations, or as guaranteeing "hallucination-free" legal citations (LexisNexis), all in 2023. It defined a hallucination to include "a false assertion that a source supports a proposition", so a real, linked case cited for the wrong point counts.
A sighting
The study ran more than 200 preregistered questions through Lexis+ AI, Thomson Reuters's Ask Practical Law AI and Westlaw's AI-Assisted Research, and through GPT-4 for comparison. The questions to Lexis+ AI, Ask Practical Law AI and GPT-4 were run between 22 March and 22 April 2024; those to Westlaw's tool between 23 and 27 May 2024. The published results (Journal of Empirical Legal Studies, 2025, Figure 4) were:
| Tool, as tested in 2024 | Hallucinated answers |
|---|---|
| Lexis+ AI | 17% |
| Ask Practical Law AI | 17% |
| Westlaw AI-Assisted Research | 33% |
| GPT-4 (no research database) | 43% |
Ask Practical Law AI gave incomplete answers, refusals or answers without grounding, to more than 60% of the questions. One example in the paper: Westlaw's tool said that a paragraph in the United States Federal Rules of Bankruptcy Procedure made certain deadlines jurisdictional, and no such paragraph exists.
The limits matter. The questions were on United States law, put to the United States versions of the tools, and the tools have changed since. The paper itself says that, after its testing, LexisNexis released a "second generation" version of its tool, and that its results "do not speak to the performance of this second generation product" (endnote 12). The figures describe those products in spring 2024, not what a firm would buy today.
The study reached England and Wales through the Bar Council. Its guidance cites the study, says that hallucinations "still occur" in tools like Lexis+ AI, and says it is essential that barristers "verify that any sources or authorities cited by such systems actually support the propositions asserted" (Considerations when using ChatGPT and generative artificial intelligence software based on large language models, last reviewed 25 November 2025, paragraph 11).
Why it happens
A research tool narrows the problem by fetching documents before it writes. The study found that this helps, but that "Many failures in the three systems stem from poor retrieval": the tool finds the wrong source, or misreads the right one (sections 6.2 and 6.3). Grounding an answer in a database reduces invention. It does not make the tool read the database correctly.
What to say back
"Hallucination-free in what sense: that the link opens a real document, or that the answer is right? Has anyone measured it on questions of English law, and can the firm see the results?"
Origin
- 2023. Legal research vendors describe their tools as eliminating or avoiding hallucinations, or as giving "hallucination-free" citations (as quoted in the study's abstract).
- 30 May 2024. The study is posted on arXiv.
- 14 March 2025. The study is accepted by the Journal of Empirical Legal Studies, which publishes it in 2025.
- 25 November 2025. The Bar Council's reviewed guidance cites it (paragraph 11).
- Now. On 1 October 2026, the LexisNexis UK home page described its Lexis+ with Protégé product as offering "linked hallucination-free legal citations". On its face the phrase is about citations that are linked, not about whether the answer is right. Whether the case says what the answer says is still for the reader to check.