Free preview6 min

Tokens: The Invisible Grammar

Reviewed by Human · Updated July 14, 2026

Real world

You look at a LEGO castle and see a castle. The builder sees bricks. Some walls are one big piece. The weird curved tower? Dozens of tiny bricks jammed together.

AI parallel

You look at your prompt and see words. The model sees tokens. Common words are one clean piece. Rare words, names, and numbers get jammed together from fragments — and that's where things get strange.

AI models have processed more text than any human could read in a thousand lifetimes. So — how many words does a model actually know?

What Are Tokens in AI?

**Token**: A chunk of text — often a word, part of a word, or punctuation — that an AI model treats as one unit. **Tokenization**: The process of splitting your text into tokens before the model sees it. Here's the deal. Before your prompt reaches the model, a tokenizer slices it up. Common English words usually survive as single tokens. Rare words get split into pieces. So do most names, typos, and long numbers. Why should you care? Because the model reasons over these chunks, not your words. Ask it to count the letters in a word, and it's working from pieces — not letters. Ask about an unusual name, and it's stitching fragments together. You can see this yourself with OpenAI's public tokenizer tool (https://platform.openai.com/tokenizer). Paste anything in. Watch it get sliced.

Try it now: paste your own name into platform.openai.com/tokenizer. Did it survive in one piece?

Watch a word fall apart

▎

Your turn

Play tokenizer. For each item below, guess: one token, or several pieces? Then check your guesses at platform.openai.com/tokenizer.

Reflect

Which guess surprised you most? That gap between your intuition and the tokenizer is exactly what this course closes.

A model keeps miscounting the letters in 'strawberry'. What's the most likely root cause?

So tokens are the raw material. But chunks alone don't decide meaning — the model reads every token against everything around it. That's why context is everything when the model interprets your words. Next up: why the *verb* you pick changes what you get back.

📚

Go deeper

end of module

You've finished this module.

Mark it complete to earn your XP and keep your streak alive.

Progress saved locally · Sign up to earn XP

Why 'Summarize' Works Differently Than 'Extract Key Points'→