Tokenization Workshop (TokShop) @COLM2026

tokshop.bsky.social

30 September 2026 06:48

Announcing our first invited speaker at TokShop! We're thrilled to welcome Tiago Pimentel @tpimentel.bsky.social (ETH Zürich) for a talk on: "How much does tokenisation impact language models?" 🧵

Maxime Méloux

maximemeloux.bsky.social

11 July 2026 03:54

We took 10 complex systems from various fields of science: a gas particle simulation, the MOS 6502 CPU, a Tracr-compiled transformer, etc. We paired each with a known ground-truth abstraction and handcrafted invalid contrastive versions. Then we threw 30+ validity metrics at them. 2/n

Maxime Méloux

maximemeloux.bsky.social

11 July 2026 03:54

The verdict: almost all metrics (MSE, probing, IB, RSA, Sobol…) are useful diagnostics but not validity criteria. They can't separate a right explanation from a wrong one that looks identical. Only causal abstraction metrics pass, and only when they test faithfulness over unmapped variables. 3/n

Maxime Méloux

maximemeloux.bsky.social

11 July 2026 03:54

So we introduce CAE (Causal Abstraction Error), a continuous validity score with a built-in faithfulness test. It passes every discrimination test, and scales/converges nicely. We release the library and the benchmark: pip install causal-abstraction-eval 5/n

Maxime Méloux

maximemeloux.bsky.social

11 July 2026 03:54

Come find us at the poster (board 125, morning session) to discuss in more detail! 📄 melouxm.github.io/CAE-web 💻 github.com/MelouxM/CAE w/ @tpimentel.bsky.social, François Portet and @peyrardmax.bsky.social