Spotting a confidently wrong answer before it burns you

A lawyer filed a brief packed with real-looking case citations, all invented by AI, and didn't catch a single one before it landed in court.

In federal courts, lawyers have been sanctioned, fined, and publicly reprimanded for filing briefs full of AI-fabricated case citations. Complete with names, docket numbers, page references. The model wrote them with the same confidence it writes everything. The lawyer read them, recognized the format, and signed.

The citations were not real.

That's the trap. An AI hallucination doesn't announce itself. The wrong answer looks exactly like the right one.

This piece is about field identification: the patterns that tend to show up when a model is filling in gaps with invention. Once you can recognize the patterns, the verification habit is the next step. And if you want to understand why the model is built this way in the first place, why AI sounds confident when it's wrong covers the mechanism.

The danger in one line

A wrong answer from an AI reads exactly like a right one.

There is no red flag, no hedged tone, no "I'm not totally sure here." A large language model is a next-word predictor optimized for fluent, plausible text. Studies have found it often sounds more confident when wrong, not less. The confident tone is a style output, not a reliability signal. Which means you can't use tone to catch it.

What you can use are patterns. Not foolproof, but worth knowing.

Six patterns that should make you look twice

These are the places where fabricated output tends to show up. Not theoretical. Operator-relevant.

Fabricated citations and named sources.

Study titles, case names, book page numbers, regulation numbers. The more specific and authoritative a citation looks, the more it earns your trust. That's exactly the problem. A real citation survives a click. A fabricated one usually does not. If there's a named source, open it. The source has to exist and say what the model says it says.

Suspiciously precise or suspiciously round numbers.

"73.4% of mid-market CFOs" with no named source. Or a too-clean "exactly 2x." Specificity is not accuracy. Ask: where would this number actually come from? If you cannot name a plausible source, treat it as invented.

Confident claims about your own private business.

The model cannot know your real revenue, your customers, or what your team decided last quarter. If it states those as fact and you never gave it those facts, it is filling a gap with invention. Watch for output that paraphrases what you told it, then quietly adds detail you never provided.

Invented quotes and attributions.

"As Bezos said..." or a line attributed to a named person. Quotes feel verifiable because they carry a source. They rarely get checked. If a quote is doing real work in your memo or deck, it needs a primary source, not a model's confident rendition.

Confident claims about recent events, prices, or releases.

Models have a training cutoff. After it, they extrapolate from older patterns in the same assured tone. Current pricing, a new regulation, a product released six months ago: these are high-risk zones. The output will sound current. It may not be.

A smooth, tidy answer to a genuinely contested question.

Real complexity produces hedging and trade-offs. If a hard, actively debated question gets a clean two-paragraph answer with no "it depends," that smoothness is the flag. Contested questions don't resolve neatly. When they look like they do, the model is probably papering over real uncertainty.

The one triage question

Before you spend five minutes verifying something, ask yourself: "Would I stake real money or my reputation on this if it turns out to be wrong?"

If yes, verification is not optional. If no, and the stakes are low and reversible, ship it and move on.

The habit is not to check everything. That's slower than doing the research yourself. The habit is to check the things that can hurt you, and to know which things those are.

Confidence is not a signal

This is worth saying plainly, because it's counterintuitive.

The model is trained to produce fluent, authoritative prose. Studies have found it often sounds more confident when wrong, not less.

The same well-formatted, hedging-free output you get when the model is right is what you get when it's guessing. The sure tone means the model is functioning as designed. It tells you nothing about whether the answer is correct.

This is not a flaw the next model release will fix. It is structural, baked into how the model works. The right move is not to look for confident answers versus uncertain ones. It's to learn where fabrication tends to show up, and check those spots.

What to do when one of these patterns fires

Stop. Ask for the source. Open it.

That's it. The 3-step verification habit takes under two minutes on any piece of output. It's worth running every time one of the six patterns above shows up in something you're about to use or send.

The first time you catch a fabricated citation before it leaves your desk, you'll understand why the lawyers who didn't wish they had.

Want to maximize your AI leverage? Upgrade to Pro.