Hallucinations, grounding, and healthy scepticism
Explain why LLMs invent plausible falsehoods and how to reduce reliance on ungrounded answers.
Sign in to track progress on this lesson.
01 — Main lesson
Full walk-through. · 8.1 MB
Spoken script — useful when names or terms sound ambiguous.
Welcome back. This is Lesson 5 in our series on large language models. Last time we talked about context windows and how the application, not the model, manages what the model sees. Today we take that idea further and look at why models sometimes produce confident nonsense, and what you can do about it. The topic is hallucinations, grounding, and healthy scepticism.
Let us start with the word itself. Hallucination. In everyday speech it suggests someone seeing things that are not there. In the world of large language models, it means the model produces text that is fluent, confident, and wrong. Not wrong in an obvious way. Wrong in a way that sounds completely plausible. A made-up citation that looks real. A function that does not exist but has a perfectly reasonable name. A fact stated with total assurance that happens to be false.
This is the thing that catches people off guard. The model sounds so sure. The grammar is perfect. The reasoning flows. The structure is clean. And yet the content can be fabricated. If you are a technical professional, this is the single most important habit to internalise. Fluency is not truth. Plausibility is not evidence.
So why does this happen? To understand hallucination, you have to remember what the model is actually doing. A large language model is trained to predict the next token. That is the core objective. Given a sequence of text, it produces a continuation that is statistically likely. The model has seen enormous amounts of text during training, and it has learned patterns of language, patterns of argument, patterns of explanation. When you ask it a question, it generates a response by continuing from your prompt in a way that matches those patterns.
Notice what is not in that description. There is no database lookup. There is no fact-checking step. There is no internal mechanism that says, is this statement actually true? The model is not retrieving from a knowledge base and then reading it back to you. It is generating, token by token, based on patterns it learned. Sometimes those patterns align with true facts, because true facts appeared frequently in the training data. Sometimes they do not. The model has no reliable way to tell the difference.
Think of it this way. The model is like a very well-read person who is trying to give you a helpful answer from memory, but who cannot look anything up, and who has an overwhelming desire to be helpful and sound confident. If you ask about something it knows well, you get a good answer. If you ask about something it half-remembers, it fills the gaps with what sounds right. If you ask about something it does not know at all, it may still produce something that sounds like an answer, because that is what it does. It generates plausible text.
This is why the objective matters so much. The training objective is plausible continuation, not truth guarantees. Nobody has yet figured out how to train a model that only says true things. You can reduce hallucinations. You cannot eliminate them with prompting alone.
Let me give you a concrete example. Suppose you ask a model to summarise a research paper, but you do not give it the paper. You just say, summarise the 2023 paper by Smith and colleagues on transformer efficiency. If that paper exists and was in the training data, you might get something reasonable. If it does not exist, or if the model is confusing it with a different paper, you will get a summary that sounds completely legitimate. It will have an abstract structure. It will mention methods and results. It will cite a finding or two. And it will be fabricated. The model is not lying. It is doing what it was trained to do, which is produce text that looks like a research summary.
Or consider code. You ask a model for a library function to do something specific. It gives you a function with a clean signature, good variable names, and sensible logic. But the function it references, or the parameter it uses, does not exist in that library. It invented it because it sounded right. This is extraordinarily common with APIs. The model knows the general shape of the library, but it fills in specifics from pattern, not from documentation.
So what do you do about it? This is where grounding comes in. Grounding is the practice of anchoring the model's response in specific, verifiable material that you provide in the context. Instead of asking the model to answer from its parametric memory, the knowledge baked into its weights, you give it the source material and ask it to answer based on that material.
Remember the image from last time. The context window is the model's working desk. Grounding is about making sure the right documents are on that desk, and then instructing the model to use them. You are not just hoping the model remembers something. You are putting the facts in front of it and saying, work from this.
This is the idea behind retrieval-augmented generation, often called RAG, which you may have encountered. The concept is simple even if the engineering is not. You retrieve relevant documents from a knowledge base, you put those documents into the context window, and then you ask the model to generate its answer based on them. Retrieve, then generate. The retrieval step is where the facts come from. The generation step is where the language comes from. By separating fact retrieval from language generation, you get something more reliable than asking the model to do both from memory.
But grounding is not just about retrieval. It is about the entire posture you take when you use the model. Let me walk you through what good grounding looks like in practice.
First, provide the source. If you want the model to answer questions about a policy document, put the policy document in the context. If you want it to analyse data, give it the data. Do not ask it to recall the policy or reconstruct the data from memory. The model is a reasoning engine over what you give it, not a database.
Second, instruct the model to stay within the source. Tell it explicitly, answer only using the information in the provided text. If the answer is not in the text, say you do not know. This instruction does not guarantee perfect behaviour, but it meaningfully reduces fabrication. Models are responsive to this kind of instruction because it shapes the pattern of generation. You are telling it what kind of response is appropriate.
Third, ask for citations. You can instruct the model to reference specific parts of the source material when it makes a claim. Quote the section, cite the paragraph, point to the line. This does two things. It makes the model's reasoning more traceable, and it gives you, the human, a way to check. But here is the critical part. Citations help only if someone actually verifies them. A model can produce a citation that looks perfect and points to text that says something different, or text that does not exist in the source at all. The citation is a pointer, not a guarantee. If the stakes are high, you follow the pointer and read the source yourself.
This is a subtle but important point. A confident citation from a model is still generated text. It is not evidence. Evidence is the source document itself. The citation is just the model telling you where it thinks it got the idea. Sometimes it is right. Sometimes it is pointing at something vaguely related. Sometimes it is pointing at nothing. You have to check.
Fourth, design for verification. If you are building an application, think about how a human will check the model's output. Can the user see the source passage that the answer is based on? Can they click through to the original? Can they compare the claim to the evidence quickly? The best grounded systems make verification easy. The worst ones bury the source and present the model's answer as if it were authoritative on its own.
Fifth, recognise when the stakes are high enough for human review. Not every use of a model needs a human in the loop. If you are drafting an email, brainstorming ideas, or exploring a design space, a hallucination is a minor inconvenience. You catch it, you fix it, you move on. But if you are generating medical advice, legal analysis, financial guidance, or anything where a wrong answer causes real harm, you need human review and you need external checks. The model is a first draft, not a final answer.
Let me give you an analogy. A large language model is like a brilliant junior colleague who has read widely, writes beautifully, and is eager to help, but who sometimes confuses things they read with things they think they read, and who cannot always tell the difference. You would not hand their first draft to a client without checking it. You would give them the source material to work from. You would ask them to cite their sources. And on anything that matters, you would review their work yourself. That is the right posture.
Now, there is a temptation to think that if you just prompt better, hallucinations will disappear. This is not quite right. Better prompting reduces hallucinations. Grounding reduces them further. But the model's fundamental nature does not change. It is still generating tokens based on patterns. Even with perfect grounding, a model can misread a source, misinterpret a nuance, or blend information from the source with patterns from its training data in ways that introduce errors. You can push the error rate down. You cannot push it to zero.
This is why healthy scepticism is not a temporary phase you go through while learning about models. It is a permanent stance. The question is not, will models stop hallucinating? The question is, have I built the right checks around the fact that they will?
Let me give you a personal checklist you can carry with you. Three questions to ask before you trust an output.
First, what is the cost of being wrong? If the model gives you a wrong answer, what happens? If the answer is nothing much, you can move fast and verify lightly. If the answer is someone gets hurt, loses money, or makes a bad decision, you slow down and verify carefully. The stakes determine the scrutiny.
Second, is this answer grounded in something I can check? If the model is answering from its own weights, with no source material provided, treat it as a suggestion, not a fact. If it is answering from a source you provided, check whether the answer actually matches the source. If you cannot check, do not trust.
Third, do I have the expertise to catch a hallucination here? Models are most dangerous in domains where you cannot tell the difference between a good answer and a plausible-sounding wrong one. If you are working outside your own expertise, be extra careful. Ask someone who does know. Or find a source you trust and verify against it. The model is not a substitute for expertise you do not have. It is a tool that works best when you have enough expertise to catch its mistakes.
And one more question, the most important one. Should I be using a large language model for this at all? There are tasks where a model is the wrong tool. Tasks that require guaranteed correctness. Tasks where you need a provable answer, not a probable one. Tasks where the downside of a single error is catastrophic and there is no human review step. In those cases, the honest answer is that a language model is not the right instrument. It does not matter how well you prompt it. The architecture does not provide the guarantee you need.
This is not anti-model. It is pro-clarity. Knowing what a tool can and cannot do is the mark of someone who uses it well. The model is extraordinary at language tasks, at synthesis, at drafting, at exploration, at transforming information from one shape to another. It is unreliable as an unattended oracle. Hold both of those facts at the same time and you will use it well.
So here is the takeaway from this walk. Hallucinations are not a bug you can patch away with a clever prompt. They are a consequence of what the model fundamentally is, a next-token predictor trained for plausible continuation. Grounding is your primary defence. Put the right material on the desk, instruct the model to use it, ask for citations, and then verify those citations yourself. Match your level of scrutiny to the stakes. And when the stakes are too high for any error, recognise that a language model alone is not sufficient.
On the next walk, we will look at prompting in more depth, including chain-of-thought and extended thinking, which are techniques for getting the model to show its reasoning steps before it gives you an answer. This is relevant to hallucinations too, because visible reasoning gives you more surface area to check. But that is for next time. For now, let this settle. Fluency is not truth. Grounding is your friend. Scepticism is permanent.
Let us start with the word itself. Hallucination. In everyday speech it suggests someone seeing things that are not there. In the world of large language models, it means the model produces text that is fluent, confident, and wrong. Not wrong in an obvious way. Wrong in a way that sounds completely plausible. A made-up citation that looks real. A function that does not exist but has a perfectly reasonable name. A fact stated with total assurance that happens to be false.
This is the thing that catches people off guard. The model sounds so sure. The grammar is perfect. The reasoning flows. The structure is clean. And yet the content can be fabricated. If you are a technical professional, this is the single most important habit to internalise. Fluency is not truth. Plausibility is not evidence.
So why does this happen? To understand hallucination, you have to remember what the model is actually doing. A large language model is trained to predict the next token. That is the core objective. Given a sequence of text, it produces a continuation that is statistically likely. The model has seen enormous amounts of text during training, and it has learned patterns of language, patterns of argument, patterns of explanation. When you ask it a question, it generates a response by continuing from your prompt in a way that matches those patterns.
Notice what is not in that description. There is no database lookup. There is no fact-checking step. There is no internal mechanism that says, is this statement actually true? The model is not retrieving from a knowledge base and then reading it back to you. It is generating, token by token, based on patterns it learned. Sometimes those patterns align with true facts, because true facts appeared frequently in the training data. Sometimes they do not. The model has no reliable way to tell the difference.
Think of it this way. The model is like a very well-read person who is trying to give you a helpful answer from memory, but who cannot look anything up, and who has an overwhelming desire to be helpful and sound confident. If you ask about something it knows well, you get a good answer. If you ask about something it half-remembers, it fills the gaps with what sounds right. If you ask about something it does not know at all, it may still produce something that sounds like an answer, because that is what it does. It generates plausible text.
This is why the objective matters so much. The training objective is plausible continuation, not truth guarantees. Nobody has yet figured out how to train a model that only says true things. You can reduce hallucinations. You cannot eliminate them with prompting alone.
Let me give you a concrete example. Suppose you ask a model to summarise a research paper, but you do not give it the paper. You just say, summarise the 2023 paper by Smith and colleagues on transformer efficiency. If that paper exists and was in the training data, you might get something reasonable. If it does not exist, or if the model is confusing it with a different paper, you will get a summary that sounds completely legitimate. It will have an abstract structure. It will mention methods and results. It will cite a finding or two. And it will be fabricated. The model is not lying. It is doing what it was trained to do, which is produce text that looks like a research summary.
Or consider code. You ask a model for a library function to do something specific. It gives you a function with a clean signature, good variable names, and sensible logic. But the function it references, or the parameter it uses, does not exist in that library. It invented it because it sounded right. This is extraordinarily common with APIs. The model knows the general shape of the library, but it fills in specifics from pattern, not from documentation.
So what do you do about it? This is where grounding comes in. Grounding is the practice of anchoring the model's response in specific, verifiable material that you provide in the context. Instead of asking the model to answer from its parametric memory, the knowledge baked into its weights, you give it the source material and ask it to answer based on that material.
Remember the image from last time. The context window is the model's working desk. Grounding is about making sure the right documents are on that desk, and then instructing the model to use them. You are not just hoping the model remembers something. You are putting the facts in front of it and saying, work from this.
This is the idea behind retrieval-augmented generation, often called RAG, which you may have encountered. The concept is simple even if the engineering is not. You retrieve relevant documents from a knowledge base, you put those documents into the context window, and then you ask the model to generate its answer based on them. Retrieve, then generate. The retrieval step is where the facts come from. The generation step is where the language comes from. By separating fact retrieval from language generation, you get something more reliable than asking the model to do both from memory.
But grounding is not just about retrieval. It is about the entire posture you take when you use the model. Let me walk you through what good grounding looks like in practice.
First, provide the source. If you want the model to answer questions about a policy document, put the policy document in the context. If you want it to analyse data, give it the data. Do not ask it to recall the policy or reconstruct the data from memory. The model is a reasoning engine over what you give it, not a database.
Second, instruct the model to stay within the source. Tell it explicitly, answer only using the information in the provided text. If the answer is not in the text, say you do not know. This instruction does not guarantee perfect behaviour, but it meaningfully reduces fabrication. Models are responsive to this kind of instruction because it shapes the pattern of generation. You are telling it what kind of response is appropriate.
Third, ask for citations. You can instruct the model to reference specific parts of the source material when it makes a claim. Quote the section, cite the paragraph, point to the line. This does two things. It makes the model's reasoning more traceable, and it gives you, the human, a way to check. But here is the critical part. Citations help only if someone actually verifies them. A model can produce a citation that looks perfect and points to text that says something different, or text that does not exist in the source at all. The citation is a pointer, not a guarantee. If the stakes are high, you follow the pointer and read the source yourself.
This is a subtle but important point. A confident citation from a model is still generated text. It is not evidence. Evidence is the source document itself. The citation is just the model telling you where it thinks it got the idea. Sometimes it is right. Sometimes it is pointing at something vaguely related. Sometimes it is pointing at nothing. You have to check.
Fourth, design for verification. If you are building an application, think about how a human will check the model's output. Can the user see the source passage that the answer is based on? Can they click through to the original? Can they compare the claim to the evidence quickly? The best grounded systems make verification easy. The worst ones bury the source and present the model's answer as if it were authoritative on its own.
Fifth, recognise when the stakes are high enough for human review. Not every use of a model needs a human in the loop. If you are drafting an email, brainstorming ideas, or exploring a design space, a hallucination is a minor inconvenience. You catch it, you fix it, you move on. But if you are generating medical advice, legal analysis, financial guidance, or anything where a wrong answer causes real harm, you need human review and you need external checks. The model is a first draft, not a final answer.
Let me give you an analogy. A large language model is like a brilliant junior colleague who has read widely, writes beautifully, and is eager to help, but who sometimes confuses things they read with things they think they read, and who cannot always tell the difference. You would not hand their first draft to a client without checking it. You would give them the source material to work from. You would ask them to cite their sources. And on anything that matters, you would review their work yourself. That is the right posture.
Now, there is a temptation to think that if you just prompt better, hallucinations will disappear. This is not quite right. Better prompting reduces hallucinations. Grounding reduces them further. But the model's fundamental nature does not change. It is still generating tokens based on patterns. Even with perfect grounding, a model can misread a source, misinterpret a nuance, or blend information from the source with patterns from its training data in ways that introduce errors. You can push the error rate down. You cannot push it to zero.
This is why healthy scepticism is not a temporary phase you go through while learning about models. It is a permanent stance. The question is not, will models stop hallucinating? The question is, have I built the right checks around the fact that they will?
Let me give you a personal checklist you can carry with you. Three questions to ask before you trust an output.
First, what is the cost of being wrong? If the model gives you a wrong answer, what happens? If the answer is nothing much, you can move fast and verify lightly. If the answer is someone gets hurt, loses money, or makes a bad decision, you slow down and verify carefully. The stakes determine the scrutiny.
Second, is this answer grounded in something I can check? If the model is answering from its own weights, with no source material provided, treat it as a suggestion, not a fact. If it is answering from a source you provided, check whether the answer actually matches the source. If you cannot check, do not trust.
Third, do I have the expertise to catch a hallucination here? Models are most dangerous in domains where you cannot tell the difference between a good answer and a plausible-sounding wrong one. If you are working outside your own expertise, be extra careful. Ask someone who does know. Or find a source you trust and verify against it. The model is not a substitute for expertise you do not have. It is a tool that works best when you have enough expertise to catch its mistakes.
And one more question, the most important one. Should I be using a large language model for this at all? There are tasks where a model is the wrong tool. Tasks that require guaranteed correctness. Tasks where you need a provable answer, not a probable one. Tasks where the downside of a single error is catastrophic and there is no human review step. In those cases, the honest answer is that a language model is not the right instrument. It does not matter how well you prompt it. The architecture does not provide the guarantee you need.
This is not anti-model. It is pro-clarity. Knowing what a tool can and cannot do is the mark of someone who uses it well. The model is extraordinary at language tasks, at synthesis, at drafting, at exploration, at transforming information from one shape to another. It is unreliable as an unattended oracle. Hold both of those facts at the same time and you will use it well.
So here is the takeaway from this walk. Hallucinations are not a bug you can patch away with a clever prompt. They are a consequence of what the model fundamentally is, a next-token predictor trained for plausible continuation. Grounding is your primary defence. Put the right material on the desk, instruct the model to use it, ask for citations, and then verify those citations yourself. Match your level of scrutiny to the stakes. And when the stakes are too high for any error, recognise that a language model alone is not sufficient.
On the next walk, we will look at prompting in more depth, including chain-of-thought and extended thinking, which are techniques for getting the model to show its reasoning steps before it gives you an answer. This is relevant to hallucinations too, because visible reasoning gives you more surface area to check. But that is for next time. For now, let this settle. Fluency is not truth. Grounding is your friend. Scepticism is permanent.
02 — Refresh
Short recap. · 856 KB
Spoken script — useful when names or terms sound ambiguous.
Quick recap of Lesson 5. Hallucination is when a model produces fluent, confident text that is not true or not supported by evidence. This happens because the model's training objective is plausible continuation, not truth. It predicts the next token based on patterns, and it has no internal fact-checking mechanism. Fluency and plausibility are not evidence of accuracy. Grounding is the practice of putting verifiable source material into the context and instructing the model to answer from it. This is the idea behind retrieval-augmented generation, where you retrieve relevant documents first, then generate. Citations help, but only if someone actually checks them against the source. A model can produce a citation that looks perfect but points to text that says something different, or to text that does not exist. For high-stakes uses, you need human review and external checks. The model is a first draft, not a final answer. Your personal checklist has three questions. What is the cost of being wrong? Is this answer grounded in something I can check? Do I have the expertise to catch a hallucination in this domain? And the most important question of all, should I be using a large language model for this task at all? Healthy scepticism is not a phase. It is a permanent stance.