A language model does not read letters or words. It reads tokens, chunks of text a few characters long, and every price, rate limit and memory ceiling is counted in them. Type anything below and four tokenizers chop it in front of you. They rarely agree.
cooking · work in progressEach tokenizer is a vocabulary, a list of chunks it knows, plus a rule for splitting text into them. The first two load on their own. The other two are a click away. The download is the vocabulary itself, which is why a bigger one costs more megabytes.
Every coloured chip is one token. A dashed chip is a single character that took more than one token, which happens when a vocabulary has never seen the letter and has to spell it out in bytes.
When the box holds more than one line, each line is counted on its own. Green is the cheapest line for that tokenizer, red the dearest.
Prices are quoted per million tokens, which sounds enormous and is not. Set a price and see what this text costs, once and a thousand times over. The dial goes from the cheapest small models to the dearest large ones; prices move often, so set it to whatever you are actually paying.
A vocabulary is built from the text a model was trained on. Common chunks get one token each, rare chunks are stitched from smaller pieces, and anything the vocabulary has never seen falls back to raw bytes, two or three of them per character outside the Latin alphabet. So the bill for a sentence depends less on the sentence than on what the tokenizer grew up reading.
That is why English is nearly always the cheapest language. Try the five-language preset: on the 2019 vocabulary the Hindi line costs about seven times the English one, on the newest between four and five. The difference is entirely down to how much of each language was in the training set when the vocabulary was built. The odd one out is the oldest: BERT spells Hindi out letter by letter and comes out cheapest on that line, because its vocabulary carries the letters of many scripts one at a time, and paying one token a letter beats paying three bytes a letter.
Watch a phone number. One tokenizer chunks the digits unpredictably, another insists on one token per digit. The second looks wasteful and is deliberate: a model that always sees digits one at a time is far better at arithmetic than one that sees 303 as a single unit one day and 30 plus 3 the next. Rare surnames are spelt out in fragments for the same reason common words are not, and one of the four lowercases everything before it starts, so it cannot tell a name from a noun.
A short paragraph is a few hundred tokens. A conversation with a model carries its whole history every turn, so a long one costs its entire length again each time you speak. The counter above says how many copies of your text fit in a million; for most texts it is thousands, not millions, and for a long document it is dozens.
These four are published as files anyone may download, and they span the algorithms and vocabulary sizes in use. Some commercial models publish their tokenizer and some do not, so the exact count a given service will bill you is only knowable from that service. The pattern is the same everywhere; the numbers differ by a little.
Counted, never judged. Nothing here says which tokenizer is better; a bigger vocabulary spends fewer tokens and more megabytes, and a model has to learn every entry in it.
Tokens are the step before meaning. What a model does with them next, turning a sentence into a position, is the embeddings tile.