Tokenizer lab / 001
Same words. Different pieces.
See how a vocabulary changes the way text is split. A from-scratch byte-pair tokenizer, a GPT-2 reference, and every byte accounted for.
32,000CUSTOM VOCABULARY / BPE
GPT-2 comparison unavailable: its original vocabulary and merge files could not be downloaded in this build environment. Custom counts are real; missing reference counts are shown as —.
01 / The playground
EDIT TEXT TO EXPLORECustom BPE 32K
11tokens
The tokenizer pipeline preserves every byte while reducing repeated sequences.
GPT-2 50,257
—tokens
Reference tables not installed.
The comparison will activate when the original GPT-2 assets are available.
The comparison will activate when the original GPT-2 assets are available.
Custom roundtrip: losslessGPT-2 roundtrip: unavailableHover a piece to see its token ID
02 / A little test corpus
20 EXAMPLES · CLICK TO LOAD| Example | Domain | Text preview | Custom | GPT-2 | Savings |
|---|---|---|---|---|---|
| Prose | The tokenizer pipeline preserves every byte while reducing repeated sequences. | 11 | — | — | |
| Code | const repository = await pipeline.process({ enabled: true, retries: 3 }); | 16 | — | — | |
| Markdown | ## observability - Preserve exact input - Measure token counts - Verify the roundtrip | 18 | — | — | |
| Data | {"component":"configuration","enabled":true,"latency_ms":42,"region":"eu-west"} | 21 | — | — | |
| Unicode | authentication: café, naïve, résumé — こんにちは 🌿 Every byte matters. | 15 | — | — | |
| Prose | The serialization pipeline preserves every byte while reducing repeated sequences. | 11 | — | — | |
| Code | const deployment = await pipeline.process({ enabled: true, retries: 3 }); | 16 | — | — | |
| Markdown | ## infrastructure - Preserve exact input - Measure token counts - Verify the roundtrip | 18 | — | — | |
| Data | {"component":"accessibility","enabled":true,"latency_ms":42,"region":"eu-west"} | 21 | — | — | |
| Unicode | synchronization: café, naïve, résumé — こんにちは 🌿 Every byte matters. | 15 | — | — | |
| Prose | The documentation pipeline preserves every byte while reducing repeated sequences. | 11 | — | — | |
| Code | const visualization = await pipeline.process({ enabled: true, retries: 3 }); | 16 | — | — | |
| Markdown | ## performance - Preserve exact input - Measure token counts - Verify the roundtrip | 18 | — | — | |
| Data | {"component":"transaction","enabled":true,"latency_ms":42,"region":"eu-west"} | 21 | — | — | |
| Unicode | notification: café, naïve, résumé — こんにちは 🌿 Every byte matters. | 15 | — | — | |
| Prose | The architecture pipeline preserves every byte while reducing repeated sequences. | 11 | — | — | |
| Code | const compression = await pipeline.process({ enabled: true, retries: 3 }); | 16 | — | — | |
| Markdown | ## collaboration - Preserve exact input - Measure token counts - Verify the roundtrip | 18 | — | — | |
| Data | {"component":"application","enabled":true,"latency_ms":42,"region":"eu-west"} | 21 | — | — | |
| Unicode | transformation: café, naïve, résumé — こんにちは 🌿 Every byte matters. | 15 | — | — |
Mixed prose, code, markdown, structured data & Unicode.Savings pending GPT-2 reference tables
01 / A shared engine
One byte-level BPE encoder and decoder. The vocabulary and ordered merges are the only moving parts.
02 / Trained before you arrive
32,000 tokens learned offline from a reproducible synthetic corpus. These examples share the training domain; they are not an independent benchmark.
03 / Every byte comes back
Token IDs decode to the original UTF-8 bytes. Colors show boundaries; a Unicode character may span multiple tokens.