Preventing AI Hallucinations in Legal Work: A Verification Protocol

Courts have sanctioned lawyers for filing briefs that cited cases that do not exist, and those headlines made “AI hallucination” a familiar phrase. Fabricated citations, though, are only the most visible failure. In litigation support the quieter ones are often more dangerous: a chronology entry with the wrong date, a summary that attributes a message to the wrong parent, a quotation that is almost what the document says.

This guide is for attorneys and litigation staff who use generative AI, or supervise people who do. It explains why hallucinations happen, catalogs the forms they take in legal work, and lays out prevention controls and a verification protocol that hold up under scrutiny. The professional duties referenced come from the ABA Model Rules, ABA Formal Opinion 512 and the Federal Rules of Civil Procedure; your state’s versions and local rules may differ.

What is an AI hallucination?

An AI hallucination is output that reads as fluent and confident but is not supported by the source material or by reality. It is not a glitch; it is a natural byproduct of how generative models work.

Large language models generate text by predicting likely next words based on patterns learned in training. They are optimized to produce plausible language, not to check facts. When a model lacks the right information, it rarely says so; it produces something that fits the pattern. Tools that first retrieve real documents and then answer from them reduce the problem but do not eliminate it, because the model can still misread, misattribute or overstate what it retrieved. The practical implication: no prompt makes a model reliably truthful. Reliability comes from process.

Why fluent errors are hard to catch

Human reviewers are used to treating polish as a signal of care. A messy draft invites scrutiny; a clean, well-organized one invites skimming. AI output is almost always clean, so its errors arrive dressed as finished work. They also tend to be locally plausible: an invented case has a believable name and citation format, and a wrong date falls in the right month. The same quality that makes AI drafts feel useful makes their mistakes easy to read past. That is why verification has to be a structured task with specific things to check, rather than a read-through for anything that “looks wrong.”

What kinds of hallucinations show up in legal work?

They fall into two families: errors about the law and errors about the facts. Most firms’ AI policies address the first and overlook the second.

Why do hallucinations matter ethically?

Because the lawyer is responsible for every word that reaches the court, regardless of who or what drafted it. Several duties converge here:

• Federal Rule of Civil Procedure 11(b) and state counterparts: signing a filing certifies that legal contentions are warranted and factual contentions have evidentiary support, after a reasonable inquiry.
• Model Rules 3.1, 3.3 and 8.4(c): meritorious claims, candor to the tribunal and honesty.
• Model Rule 1.1: competence, which ABA Formal Opinion 512 reads to require understanding a tool’s limitations and verifying its output appropriately.
• Model Rules 5.1 and 5.3: supervision of the lawyers and staff who use AI tools.

Some judges and courts also require disclosure or certification of AI use in filings, so check standing orders. For the full opinion, see our guide to ABA Formal Opinion 512.

How do you prevent hallucinations before they happen?

Design the work so the model has less room to invent and every output is easy to check. Prevention reduces the volume of errors; verification catches the rest.

• Ground the tool in your record. Give it the actual documents and ask it to work only from them, rather than relying on its general knowledge.
• Require a source for every assertion. Ask for a document reference, page or message for each statement, so unsupported claims stand out.
• Narrow the task. “Extract every message from March that mentions the school pickup, with timestamps” is safer than “summarize what happened.”
• Invite “not found.” Tell the tool explicitly that saying information is absent is an acceptable answer.
• Ask for exact quotes with locations instead of paraphrases when the wording matters.
• Keep AI out of exact steps. Numbering, pagination, Bates stamping and exhibit assembly should be deterministic.
• Label AI-assisted content in drafts so reviewers know where to look hardest.

Choosing tools built this way helps. CourtBriefly, for instance, uses AI only for assistive steps like extracting text from images, which a person reviews and corrects, while its exhibit PDFs are generated deterministically without AI (details on how evidence extraction and export work).

A verification protocol for AI-assisted drafts

Verification should be a defined step with a named owner and a record, not a general instruction to “double-check.” A protocol that works for most filings:

1. Mark the AI-assisted content. The drafter flags which sections, citations and facts came from or passed through an AI tool.
2. Verify every authority. Pull each cited case, statute and rule from a reliable research source. Confirm it exists, the citation is correct, it says what the draft claims, quotations are exact, pin cites are right, and it remains good law.
3. Trace every fact. Each factual assertion gets a record cite, and the verifier opens the cited document to confirm it.
4. Check dates, names and numbers separately. These are where small errors do large damage; check them as their own pass.
5. Add a second reviewer for high-stakes filings. Dispositive motions, declarations and anything sworn deserve another set of eyes.
6. Log the verification. Record who checked what, when, and what was corrected.

What red flags suggest a passage needs a closer look?

Some patterns appear again and again in hallucinated output. Train reviewers to slow down when they see them:

• A case nobody on the team has heard of that supports the point perfectly.
• Quotations that are unusually on point or tidy.
• Citation formats that look slightly off, such as odd reporter abbreviations or page numbers.
• Summaries or fact sections with no record cites.
• Specific-sounding details, such as times, amounts or names, that you do not remember from the documents.
• Uniform confidence throughout, with no hedging where the record is actually unclear.
• Events in a chronology that fall in a suspiciously neat order.

How do you train a team to catch hallucinations?

Make the failure visible. Abstract warnings do less than seeing a fluent, wrong output on your own kind of material.

• Run a short session where staff verify a deliberately flawed AI draft and compare what they caught.
• Publish the verification protocol and log template where everyone uses them.
• Make the verifier’s name part of every filing checklist.
• Revisit the policy when tools change, and share near-misses openly so others learn from them.

For staff-level guidance, see using AI as a paralegal: supervision, verification and ethics, and for review-specific controls, AI-assisted document review ethics. The goal is not to avoid AI but to build a process in which its mistakes are caught by people before they reach a client or a court. If a question turns on your jurisdiction’s disclosure rules, check with your state bar or a licensed attorney where you practice.