Tokens, vocabulary, and why wording matters

Welcome back. This is the second walk in our series on large language models. Last time we settled on the framing that a large language model is a next-token predictor trained at enormous scale. Today we look at what a token actually is, because understanding tokens changes how you write prompts and how you think about cost, context, and clarity.

Let us start with something you already know from everyday life. When you read the word unhelpful, you do not read it as eight separate letters. You see a chunk. You see help in the middle. Your brain breaks words into meaningful pieces without you noticing. A large language model does something similar, but the pieces it uses are called tokens, and they are not always the pieces you would expect.

A token is a small unit of text. Sometimes a token is a whole word. The word cat is probably one token. The word dog is probably one token. But longer or less common words are often split into several tokens. The word unhelpful might be three tokens, something like un, help, and ful. The model does not see words the way you do. It sees a sequence of these smaller pieces, and it learns statistical patterns across them.

Here is a concrete way to picture it. Imagine you are reading a sentence one syllable at a time rather than one word at a time. You would still understand the sentence, but your experience of it would be different. You would notice boundaries differently. You might stumble on unusual combinations. That is close to what the model experiences. It works with tokens, and tokens do not always line up with word boundaries.

Why does this matter to you? Three reasons. First, it affects cost. If you are using an application programming interface that charges per token, then the way you phrase your prompt and the way the model phrases its answer both have a price tag. A prompt that uses ten words might cost more or fewer tokens than you expect, depending on how those words split. Second, it affects context. Third, it affects behaviour, and this is the one that surprises people.

Let us take context first. Every model has a context window, which is the maximum number of tokens it can hold in its working memory at once. Think of it like a desk. You can only spread out so many pages before you run out of room. When you send a prompt to a model, everything in that prompt counts toward the window. Your instructions, the conversation history, any documents you pasted in, and the model's own response so far. All of it occupies tokens on the desk.

When a conversation gets long, earlier messages may fall off the desk. The model does not forget them the way a human forgets. It simply stops receiving them as input. If you have ever had a long chat with a model and noticed it started ignoring something you said twenty turns ago, this is likely why. The context window filled up, and the system had to drop older content to make room for newer content. This is not a personality flaw in the model. It is a physical limit on how many tokens can be processed in one pass.

Now, here is where tokenisation gets quirky, and this is worth slowing down for. Not all text splits into tokens the same way. Common English words are usually efficient. A word like the or and or because is likely a single token. But numbers can behave strangely. A long number like one two three four five six seven eight might be split into several tokens, not because it is long in characters but because the tokeniser was built around word fragments and numbers do not always break cleanly. Code is even more unpredictable. Programming symbols, variable names, and indentation can all consume more tokens than you would guess. If you paste a large block of code into a prompt, you may be burning through your context window faster than you realise.

Rare names are another classic trap. If your name is common in the training data, it might be one token. If your name is rare or uses an unusual combination of characters, it might be three or four tokens. The same applies to company names, product names, and technical jargon that the tokeniser has never seen before. The model can still work with these, but each rare name eats more of your context budget.

So far this is mostly about cost and capacity. But there is a deeper point about behaviour, and it connects directly to how you should prompt.

Remember, the model predicts the next token, then the next, then the next. It is not thinking ahead in the way a human plans a paragraph. It is making one prediction at a time, each one conditioned on everything that came before. This means the exact wording of your prompt shapes the statistical landscape the model is navigating. Change a few words and you change the probabilities. This is not a minor effect. It is the core mechanism.

Here is an example. Suppose you write, explain how a database index works. The model starts generating tokens. The first few tokens it produces constrain what comes next. If it begins with the words a database index is, it is now in a definitional register. It will probably continue in a textbook style. If instead it begins with imagine a book, it is now in an analogy register. The opening tokens pull the rest of the response in a direction. You influenced those opening tokens through your wording, even if you did not control them directly.

This is why prompt clarity beats clever jargon. Some people try to engineer prompts with dense, cryptic instructions, as if the model will reward cleverness. It will not. The model is not impressed. It is doing statistical pattern completion. Clear, direct, specific language produces more predictable patterns. Vague or overly clever language produces noisier patterns. If you want a simple explanation, say so plainly. If you want a table, say table. If you want the answer in under one hundred words, say that. Each precise instruction narrows the space of likely completions.

Think of it like giving directions to a stranger in a city. If you say take me somewhere nice, you might end up anywhere. If you say take me to a quiet cafe within ten minutes walk that serves vegetarian food, you have narrowed the options enormously. The stranger is not judging your eloquence. They are just working with the constraints you gave them. A language model is that stranger, except the constraints are tokens and the city is a probability distribution over text.

There is a subtlety here that catches technical professionals. You might assume that because you understand a concept well, the model will understand your prompt the same way you mean it. But the model does not have your context. It only has the tokens you sent. If you write something terse and assume the model will fill in your intent, you are leaving the interpretation to chance. The model will fill in something, but it might not be what you assumed. Explicit is better than implicit, every time.

Let me give you one more practical example. If you write summarise this, the model has to guess what kind of summary you want. Bullet points or prose? One sentence or three paragraphs? Technical or plain English? It will pick something based on patterns in its training data, and you might not like the choice. If you instead write summarise this in three bullet points for a non-technical audience, you have removed the guesswork. The model still has freedom in how it phrases each bullet, but the shape of the output is constrained. You spent a few extra tokens to save yourself a regeneration and a lot of frustration.

This is the mental model I want you to carry. You are steering a next-token machine. The words you choose are not just a message to be understood. They are the steering input. Precision of language is a control surface. When you change your wording, you are not merely rephrasing a request. You are changing the probability distribution over what comes next. That is why the same idea expressed in different words can produce noticeably different answers, and different costs, and different token counts.

One last point before we wrap up. Do not overthink tokenisation to the point of paralysis. You do not need to count tokens manually for every prompt. What you need is a feel for it. Common words are cheap. Long documents are expensive. Rare names and code are surprisingly expensive. Long conversations fill the window and push out earlier context. And clear, specific prompts produce more reliable behaviour than clever, dense ones. If you internalise those five facts, you will prompt better than most people who have read a dozen tutorials.

On the next walk, we will look at training versus inference, because understanding the difference between how a model is built and how it is used will clarify what it can and cannot do in the moment. For now, let this idea settle. You are steering a next-token machine, so precision of language is a control surface. Say what you mean, plainly and specifically, and the model will follow you more reliably than if you try to be clever.

Quick recap. A token is a small unit of text, not always a whole word. Common words like cat or the are often one token, but longer or less common words get split into pieces. The model predicts one token at a time, repeatedly, to build its answer.

Context length is measured in tokens. Everything in your prompt, the conversation history, pasted documents, and the model's own output, all count toward the context window. When the window fills, older content drops out, which is why long chats sometimes lose earlier context.

Tokenisation has quirks. Numbers, code, and rare names can consume more tokens than you expect. A long block of code might eat through your context budget faster than plain English prose. This matters for cost and for capacity.

The key behavioural insight is this. The exact wording of your prompt shapes the probabilities of what the model generates next. Change your words and you change the likely output. This is why clear, specific prompts work better than clever, dense ones. The model is not impressed by jargon. It responds to patterns.

If you want a summary in three bullet points for a non-technical audience, say that explicitly. Do not make the model guess. You are steering a next-token machine, so precision of language is a control surface. Say what you mean, plainly and specifically.