tokenization
Also called: tokens, token, tokenizing
Tokenization is the process of splitting text into tokens, the small units (whole words, subwords, or single characters) that a language model actually reads and generates. Most modern LLMs, including GPT, use subword methods like byte-pair encoding (BPE), where roughly four characters of English map to one token.
How tokenization works
A tokenizer does not read words the way people do. It runs text through a fixed vocabulary built during training and maps each piece to an integer ID. Most large models use byte-pair encoding (BPE), a method OpenAI adopted for GPT that starts from individual characters (or bytes) and repeatedly merges the most frequent adjacent pairs until it reaches a target vocabulary size. GPT-2’s byte-level variant keeps a base of 256 byte tokens plus tens of thousands of merged tokens, so it can encode any string without an “unknown” token.
Because merges follow frequency, common words become a single token while rare or invented words split into pieces. The word “tokenization” can break into token + ization. Leading spaces, punctuation, and casing all count as part of the token. That is why the rough “four characters per token” estimate holds for plain English but falls apart for code, numbers, and other languages, which fragment into many more tokens.
Tokens are the unit everything is measured in. Context windows, API pricing, and rate limits are all counted in tokens, not words, so inefficient tokenization raises both cost and latency. You can watch exactly how a model splits your text with OpenAI’s Tokenizer tool or the tiktoken library.
How it affects your traffic
When an AI assistant answers a query using your page, it does not see your layout, it ingests your text as tokens and works inside a fixed token budget. Content broken into clean, self-contained passages tokenizes efficiently and is easier for a retrieval system to pull and quote, while walls of text, bloated markup, or sprawling tables burn tokens and can get truncated before the useful part is read. Knowing how your pages tokenize is part of getting cited in AI answers, which is the core of our AI SEO work.
Get AI SEO that moves the needle
We turn terms like this into ranked pages and qualified pipeline. Start with a free Initial SEO Strategy.