What are tokens in AI?
Fundamental units of text that AI models process, representing word pieces or characters.
In brief
- 1 token ≈ 4 characters or 0.75 words
- Unit of measurement for LLMs
- Determine context limits
Tokens are fundamental units of text that AI models process, representing word pieces, whole words, or special characters.
On average, 1 token = about 4 characters or 0.75 words in English. Tokens are used to measure prompt and response length.
Frequently asked questions
How many words is a token?
In English, a token is on average about four characters, or three quarters of a word, according to OpenAI. Other languages usually need a few more tokens for the same text. The exact split depends on the model.
Why do tokens matter?
They are the unit for context window size, maximum answer length and API billing. A long text costs more to process and may be truncated.
Should you write short for AI?
Above all, write clearly. An answer placed early on the page, without filler, is more likely to be read in full by an AI that only keeps excerpts.