How AI Chatbots Work: Tokens, Context Windows and Hallucinations

You don't need math to use ChatGPT, Claude or Gemini, but understanding how they work under the hood (the simple version) changes how you use them. It explains why they sometimes invent facts, why the same question gets different answers, why they "forget" what you said at the start of a long conversation and how to get more out of them.

The core idea: predicting the next word

A large language model (LLM) is, at heart, a system trained to predict what text comes next. If it reads "The capital of France is…", it has seen that phrase completed with "Paris" so many times that "Paris" is the most likely continuation.

The surprising part is that to predict the text of millions of books, articles, conversations and programs well, the model has to learn an enormous amount about grammar, facts about the world, reasoning, writing styles and the logic of code. Abilities that look a lot like understanding emerge from that prediction skill at massive scale.

How it's trained (in three stages)

1. Stage 1 · the most expensivePretrainingThe model reads a vast amount of public and licensed text and adjusts billions of internal parameters to get better at predicting the next word.
2. Stage 2Instruction tuningIt's trained on examples of questions and good answers so it behaves like a helpful assistant instead of just completing text.
3. Stage 3Feedback tuningPeople (and other systems) rate responses, and the model learns to prefer the ones that are most helpful, honest and safe.

To see it in action, imagine the model has to continue the sentence "On Sunday mornings I like to have coffee with…". Internally it assigns a probability to every possible continuation and picks from among the highest:

These percentages are an example to illustrate the idea, not real values from a model. They add up to 100%: the model spreads probability across every possible continuation.

Tokens: the unit it "thinks" in

Models don't read whole words; they read tokens, which are chunks of text. A common word may be one token, while a long or unusual word gets split into several. For English, a widely used rule of thumb is that 100 tokens equal about 75 words. Other languages, numbers and code usually take more tokens per word.

Tokens matter for two practical reasons:

• Length limits (on what you can send and what you get back) are measured in tokens.
• If you use AI through an API (from a program), you pay per token. You'll see real prices in lesson 10.

The context window: its working memory

The context window is the maximum amount of text the model can take into account at once: your instructions, the documents you attach, the earlier conversation and its own answer. Today's models have very large windows. The Claude models current in November 2026, for example, accept up to 1 million tokens, the equivalent of several books. But windows aren't infinite, and the fact that a model can read a lot doesn't mean it pays equal attention to everything.

Practical consequences:

• In very long conversations, information from the beginning can lose weight or fall out entirely. When you switch topics, start a new chat.
• If something matters, repeat it or summarize it in your current message.
• The model doesn't "remember" your other chats unless the app has a memory feature turned on.

Why AI "hallucinates"

A hallucination is when the model generates false information with complete confidence: a quote nobody said, a law that doesn't exist, a made-up statistic, a fake citation. It happens because the model generates the most plausible text, and an invented fact can sound very plausible.

Hallucinations are more likely when:

• You ask for very specific facts (numbers, dates, names, exact quotes, URLs).
• The topic is obscure or very recent.
• You pressure it to answer something it doesn't know ("give me 10 studies proving…").

Why the same question gets different answers

When generating each word, the model doesn't always pick the single most likely option; it adds some variation (controlled by a setting called temperature). That makes answers more natural and creative, but it means asking the same question twice gives you different versions. Use that to your advantage: if an answer doesn't work for you, regenerate it or ask for "three different versions."

Models with reasoning and tools

Today's assistants are no longer just "text predictors." Many combine the model with tools that cover its weak spots:

When the right tool is turned on, many of the classic limitations (stale facts, math errors) shrink dramatically. That's why it pays to know which features your assistant has.

Three prompts that work with how the model works

Now that you know how it "thinks," you can ask in ways that play to its strengths. These three prompts solve the most common problems:

1. To rescue a long conversation

Summarize this conversation in a block I can paste into a new
chat: the goal, decisions made, key data (with exact figures),
the format we agreed on and pending tasks. 200 words max.

Expected result: a compact summary. Open a new chat, paste it as your first message and keep going: the model has the important parts "fresh," without the noise of all the back-and-forth.

2. To make it admit what it doesn't know

Answer the question, then label each fact in your answer as
(A) certain, (B) probable or (C) I should verify this. For any
C, tell me where I could check it. Question: [your question]

Expected result: the same answer with the shaky facts flagged. It isn't foolproof (the model can label something false as "certain"), but it tells you where to start checking.

3. To put the variation to work

Give me 3 very different versions of [an email subject line /
a tagline / the opening line of my presentation]: one formal,
one friendly and one with humor. Say which audience each works
best for.

Expected result: three genuinely different options instead of one. Because the model adds variation as it generates, asking for alternatives is faster than regenerating until something clicks.

And before you use any fact an assistant gives you in something that matters, run through this list:

• Does the fact come from a document I provided or from a linked source I actually opened?
• Did I check numbers, dates, names, regulations and exact quotes against the original source?
• Is the topic very recent? If web search wasn't on, I treat it as doubtful.
• Did I ask "under pressure" (for example, "give me 10 studies that…")? That's where it invents the most.
• Could I explain and defend this fact if someone challenged it?

Privacy: what happens to what you type

Whatever you type into an assistant is processed on the servers of the company that provides it. Depending on the product and plan, conversations may be kept for a while and, in some cases, used to improve the models (there's usually a setting to turn that off). Business and enterprise plans generally offer stronger guarantees. Check your tool's privacy settings and your company's policy, and read lesson 12 before pasting anything sensitive.

Summary

• An LLM generates text by predicting the next token; its abilities emerge from that at scale.
• Tokens are chunks of text: they set length limits and API costs (about 75 English words per 100 tokens).
• The context window is its working memory: start new chats for new topics.
• Hallucinations are plausible false facts: give it sources and verify.
• Reasoning, web search and code execution greatly expand what it does well.

Next lesson: How to Write AI Prompts That Work: A Formula, Techniques and Templates. You'll learn the six-part prompt formula, seven techniques that improve any prompt and templates you can use today.