LEVEL 01 · LESSON 01
Tokens
Tokens are the units of information a model processes, not a universal word count.
What it is
Tokens are the small units a language model processes. A token can be a word, part of a word, a number fragment, or punctuation. The split depends on the model’s tokenizer and the language.
Why it matters
Tokens affect how much information fits in a request, how long an answer takes, and often what an AI service costs. Page counts and character counts are only rough substitutes.
A business example
A support team includes a long policy manual with every customer question. Most of the manual is irrelevant to each case, but it still consumes input tokens. Retrieving the relevant section can reduce this overhead while retaining the evidence needed for the answer.
How it actually works
A tokenizer converts text into numbered pieces. The model processes those pieces and predicts output tokens one after another. The application converts output tokens back into readable text. Input and output can be metered differently, and some systems also account for internal reasoning. Use the actual tokenizer or usage report when estimating a workload; rules of thumb vary substantially across languages.
A common misconception
A token is not always a word, and fewer tokens do not automatically mean a better answer. Removing a crucial exception can make a cheap answer expensive in practice.
ONE SENTENCE TO REMEMBER
Tokens are the units of information a model processes, not a universal word count.
CHECK YOUR UNDERSTANDING
A team estimates what a new AI workload will cost by counting the pages in its documents. What is the main weakness of this estimate?
Show the answer
Answer: B.
Usage is measured in tokens, and a token can be a whole word, part of a word, a number fragment, or punctuation. The split varies by tokenizer and language, and input and output are often metered differently. Estimate with the actual tokenizer or a usage report.
Take it into your next meeting
Which parts of your typical request are repeated without helping the current task?
One more concept, understood.
Mark this lesson complete to keep your place.