3/23/2026 at 10:21:14 PM
Neat. Is it a single under-trained token in GPT-5.2? Or is something else going on?by skerit
3/23/2026 at 11:31:49 PM
Perhaps, the word does have it's own token, " geschniegelt"(geschniegelt with a space in front of it), is token 192786 in the tokenizer that GPT-5 apparently uses.https://raw.githubusercontent.com/niieani/gpt-tokenizer/refs...
by WatchDog
3/24/2026 at 6:39:44 AM
Isn't giving this word a token something deeply wasteful? When some more common things are multiple tokens.Indeed, how do they deal with Chinese? Are some ideograms multiple tokens?
by nextaccountic
3/24/2026 at 7:13:30 AM
It simply means the tokenizer's training corpus may have included a massive amount of German literature or accidentally oversampled a web page where that word was frequently repeated. Look up "glitch tokens" to learn more.by mudkipdev
3/23/2026 at 11:25:22 PM
Based on their tokenizer tool[1], for GPT 5.x "geschniegelt" is tokenized into three tokens: (ges)(chn)(iegelt)
[1]: https://platform.openai.com/tokenizer
by magicalhippo
3/24/2026 at 1:15:28 AM
It's a single token in the most common usage, that is, with a space in front of it"This word is geschniegelt" is [2500, 2195, 382, 192786]
Last token here is " geschniegelt"
by Tiberium
3/24/2026 at 1:31:20 AM
Maybe this is why? Most of the training data has the single token version, so the three tokens version was undertrained?by nialv7