Marco

mcognetta.bsky.social

07 October 2026 01:14

Tokenizer research lags behind other areas because it is hard to validate changes and a lack of good predictive metrics/scaling laws. It is also just hard to compare across tokenizers. Let's have a community benchmark where everything but the tokenizer is fixed!

Federico Pianzola

fpianz.eurosky.social

30 September 2026 19:08

Extremely important for multilingual research!

Craig Schmidt

craigschmidt.com

30 September 2026 19:08

Congratulations to Marco and all the other co-authors on this groundbreaking 154-page survey. If you're at all interested in tokenization, give it a read.

Eugene Jang @ COLM

eugeneonnlp.bsky.social

30 September 2026 18:00

A huge collaborative effort to describe the deceptively deep rabbit hole that is tokenization! Hopefully it contains many answers to "has anyone tried doing ~" questions and inspire new approaches. Thank you to @mcognetta.bsky.social for leading this ambitious project!

Marco

mcognetta.bsky.social

30 September 2026 16:41

🚨 [Token][ization] Paper Alert 🚨 Tokenization is a wildly understudied area of language modeling despite it having effects across all of NLP. Over the past ~8 months, 32 (!) tokenizer researchers put together the most comprehensive survey of the field. Check it out!