Does "Do Not Hallucinate" Work? Why AI Still Makes Things Up

If you have been using AI long enough, you have probably tried some version of this:

"Answer only with facts. Do not make things up. If you do not know, say so."

It sounds reasonable. And sometimes it even feels like it works: the AI sounds more careful, hedges a bit more, adds a disclaimer here and there. But if you have ever clicked through on one of its sources, you have probably noticed the problem. The links go nowhere. The study does not exist. The statistic has no traceable origin.

The "do not hallucinate" prompt did not fix the source problem. It just made the hallucination sound more polite.

Why the prompt does not actually stop AI hallucination

AI is not deciding whether to lie to you. It is predicting what a helpful, plausible answer looks like based on patterns it learned during training. When it generates a citation, it is not necessarily looking up a real paper. It is producing text that looks like what a citation looks like.

Telling it "do not make things up" is like telling someone to stop dreaming. They cannot choose not to dream. They can only describe the dream more carefully.

A caution prompt can change the surface behavior. The model may hedge more, refuse more often, or remind you to verify. But it does not give the model a reliable source-checking mechanism.

So the AI gets more careful-sounding. It adds "according to some sources" or "you may want to verify this." But the underlying source it just invented can still be invented. The AI hallucination problem does not go away. It gets better packaging.

What actually happens in a typical scenario

Say you are reading a news story and you are not sure if it is accurate. You paste it into ChatGPT and ask: "Is this true? Give me sources."

The AI gives you a confident paragraph, two or three source links, maybe a journal name or a statistic. It sounds credible. You feel reassured.

But the answer can sound right, the citation can look right, and the connection between them can still be broken. The source links might resolve. Or they might go to a completely unrelated page. Or the URL just does not exist at all.

You cannot tell from reading the answer. That is the problem.

The part you can actually fix

You cannot prompt your way out of AI hallucination. But you can check the sources without doing it manually.

AI FactScan is a Chrome and Edge extension that checks AI-generated sources while you are still on the page. Select the text, run the scan, and it surfaces dead links, invalid citations, AI self-references, and other source-quality risks. No new tabs, no copy-pasting URLs one by one.

It will not tell you whether the source says what the AI claims it says. That still needs you. But it catches the obvious first-layer problems: the links that go nowhere, the citations that were never real, and the sources that deserve closer inspection before you trust them.

The takeaway

Adding "do not hallucinate" to your prompt is worth trying. But it changes tone and caution more reliably than it changes the underlying verification problem. The model can sound more careful without actually being more accurate.

If a source matters, check whether it exists before you trust it. That step does not have to be painful, and it does not require a better prompt.

AI FactScan is a free Chrome and Edge extension that checks AI-generated sources directly inside ChatGPT, Claude, Gemini, Perplexity, and more.