Hallucination is when an AI language model generates output that is factually incorrect, fabricated, or unsupported by available evidence, while presenting it with the same confidence as accurate information. The model is not lying or guessing in the way a person might, but it’s doing what it was designed to do, predicting the most statistically likely next token given its training and the current context. Sometimes that process produces accurate, well-grounded responses. Sometimes it produces plausible-sounding information that is simply wrong.
Hallucination is a structural characteristic of how language models work, not a bug to be patched. It’s more likely to occur when a model is asked about topics underrepresented in its training data, when it’s asked to produce specific facts like dates, figures, or citations, or when the context window lacks grounding information relevant to the query.
Why This Matters for Technical Writers
Hallucination has direct implications for how AI system outputs should be documented and how users should be prepared to interact with them. Documentation that presents AI outputs as reliable facts without qualification sets users up to overtrust a system that can be confidently wrong. Technical writers documenting AI products need to set accurate expectations about where hallucination is more likely, what users should verify independently, and how the product handles or mitigates it, whether through RAG, human review, or other mechanisms. This is especially important in high-stakes contexts like legal, medical, or financial applications where a confident but incorrect output can cause real harm.
Common Confusion
Hallucination is sometimes used as a catch-all term for any AI output a user disagrees with or finds unhelpful, which dilutes its meaning. Hallucination specifically refers to generated content that is factually incorrect or fabricated, not content that is merely incomplete, imprecise, or not what the user was looking for. It’s also sometimes treated as a solved problem in systems that use RAG or grounding techniques, but those approaches reduce hallucination rather than eliminate it.
Related Terms
Probabilistic, deterministic, RAG, large language model (LLM), token, context window, grounding, human-in-the-loop