Tokens per Word: GPT-5 vs Claude vs GPT-4, Measured (2026)

Why tokens per word decides your bill

Every large language model bills by the token, never by the word. The exchange rate between those two units is where API budgets quietly drift. Most planning guides repeat the same rule of thumb: one token is about three quarters of an English word. That figure is roughly right for English on a modern tokenizer, and increasingly wrong for everything else: other languages, source code, structured data, and emoji all convert at their own rates.

Published numbers on this are surprisingly thin, so we measured it. This article reports exact token counts for the same content across five tokenizers and three model families, with the corpus and results downloadable below. If you budget LLM usage in any language other than English, the differences are large enough to change your projections.

The dataset and how it was measured

The corpus has 13 samples. Seven are human translations of the same 94-word passage about editing, in English, Spanish, Portuguese, French, German, Chinese, and Japanese, so the cross-language comparison holds meaning constant rather than length. The other six cover the text developers actually send to models: Python, JavaScript, a JSON order record, a Markdown document, an emoji-heavy social post, and CSV numeric data.

Counts for the GPT family come from tiktoken, OpenAI's published tokenizer, so they are exact: o200k_base (GPT-5, GPT-4o, the o-series), cl100k_base (GPT-4, GPT-3.5), and the GPT-3 era p50k_base for historical contrast. Claude counts come from Anthropic's official count-tokens API endpoint, which reports the billable figure per model. The endpoint counts the whole request, so we measured the fixed message envelope (6 tokens on Opus 4.8, 7 on Sonnet 4.6 and Haiku 4.5) and subtracted it, then verified the calibration with a doubling check that came back with zero drift. Absolute Claude counts carry about one token of uncertainty; ratios are unaffected.

Gemini is excluded from the measurements because Google does not publish its tokenizer and we had no countTokens access to verify against; we would rather scope the data honestly than estimate.

Tokens per word by language

The headline table. Same passage, same meaning, five tokenizers:

English is the cheapest language in every column: 110 tokens for 94 words on GPT-5, or about 1.17 tokens per word. The popular 0.75-words-per-token rule holds almost exactly for English prose. Spanish runs 1.34 tokens per word on the same encoding, Portuguese 1.34, French 1.40, and German, with its long compounds, 1.71. Chinese and Japanese have no whitespace word boundaries, so per-word figures are not applicable; the next section compares them on equal meaning instead.

Same meaning, different price

Because all seven passages say the same thing, the fairest question is: what does it cost to express identical meaning in each language? Taking English as the baseline:

On GPT-5, expressing this passage in Spanish costs 30% more tokens than in English; Portuguese costs 25% more, and Japanese 86% more. The penalty grows on older encodings: the same Spanish passage that costs +30% on o200k cost +56% on GPT-4's cl100k, and the GPT-3 era p50k encoding needed 222 tokens for it, more than double its English equivalent. Anyone running multilingual workloads inherited those legacy ratios in their intuition, and they are now badly out of date.

The o200k effect: three GPT generations

The encoding history explains the shift. p50k and cl100k were trained heavily on English; o200k doubled the vocabulary to around 200,000 tokens and allocated far more of it to non-English text. For Spanish, the progression is 222 tokens (GPT-3 era) to 172 (GPT-4) to 143 (GPT-5) for the identical passage. Chinese improved even more sharply: 223 tokens on cl100k against 159 on o200k, a 29% drop.

The improvement is not universal. Our JavaScript sample is one honest counterexample: it costs 140 tokens on cl100k and 149 on o200k, slightly more on the newer encoding. English prose and Python were essentially flat. o200k's gains went to human languages, not to code.

Claude counts twice: Opus 4.8 vs Sonnet 4.6

The least documented result in the dataset: Anthropic's count-tokens endpoint reports two distinct counting regimes across its current models. Sonnet 4.6 and Haiku 4.5 return identical counts for every sample in the corpus. Opus 4.8 reports substantially higher figures for the same text, which matches Anthropic's own migration notes that Opus 4.7 and later count tokens differently.

The inflation is concentrated in Latin-script text, where Opus reports roughly 1.3 to 1.5 times the Sonnet count. On Chinese and Japanese the two regimes nearly coincide. This matters for budgeting because the billable unit differs by model: Opus 4.8 at $5 per million input tokens does not cost 1.67 times Sonnet 4.6 at $3 for English prose; measured end to end it costs about 2.5 times as much per word, because each word registers as more tokens. The cost table below uses each model's own measured counts.

Code, JSON, and CSV cost more than prose

Per character, structured text is far denser than prose. Punctuation, brackets, quotes, and digits fragment into many small tokens:

CSV numeric data is the most expensive input in the corpus at 57 tokens per 100 characters, three times the density of English prose. Dates, IDs, decimals, and percent signs tokenize one fragment at a time. The practical advice: when you pipe spreadsheets or logs into a model, the character count will mislead you; count tokens on a representative chunk first, and consider summarizing or sampling numeric tables before sending them whole.

Emoji are expensive

The social-media sample packs 11 emoji into 283 characters. Each emoji costs one to three tokens on o200k, and skin-tone or compound variants cost more. The sample lands at 88 GPT-5 tokens, a per-character density between prose and code. For chat products that process social text at scale, emoji are a real line item, not a rounding error.

What a million words costs

Converting measured tokens per word into input cost at current published prices (GPT-5 $1.25, GPT-5 mini $0.25, GPT-4o $2.50, Claude Haiku 4.5 $1.00, Sonnet 4.6 $3.00, Opus 4.8 $5.00 per million input tokens) gives the number a budget owner actually wants, the cost to process one million words:

Two readings of this table. First, language overhead compounds with model choice: a million German words through Opus 4.8 costs $17.42 against $1.46 for English through GPT-5, a 12x spread for the same volume of meaning. Second, input pricing is cheap everywhere in absolute terms; the ratios matter when you multiply by output tokens, which typically cost four to five times the input rate and follow similar per-language inflation.

Reproduce the numbers

The full dataset and corpus are free to download and reuse with attribution (CC BY 4.0):

• tokenizer-comparison-2026.csv, every measurement in one flat table
• tokenizer-comparison-2026.json, measurements plus methodology and derived metrics
• tokenizer-corpus-2026.json, the 13-sample corpus, so you can verify every count

To check the GPT figures, run any sample through tiktoken with the o200k_base or cl100k_base encoding. To check Claude, call Anthropic's count-tokens endpoint with the sample as a single user message and subtract the envelope as described above. To get a feel for the numbers interactively, paste any corpus sample into our browser-local Token Counter: it runs the real o200k encoding client side, so the GPT counts match this dataset exactly and your text never leaves the page. For background on what a token is in the first place, see the Token Counter complete guide.