Mathias Gehrig

mathiasgehrig.bsky.social

06 February 2025 22:22

reading the blog: it appears there is a difference between recomputing and recompiling. The latter is expensive but does not seem to be necessary even if the block-sparsity pattern changes. Indeed magic

François Fleuret

francois.fleuret.org

06 February 2025 22:21

It is hard to overstate how cool and powerful is flex attention. @chhillee.bsky.social pytorch.org/blog/flexatten… TL;DR: it is an implementation of the attention operator in pytorch that allows in particular to efficiently "carve" the attention matrix. 1/3

Alex Nichol

unixpickle.bsky.social

07 January 2025 06:32

Asking the heavy hitter :P Instead of a destination, I'd prefer to imagine a process. I think we aren't that far from something we can keep training indefinitely that will keep climbing on evals as long as we run it with no end in sight.

Birchlabs

birchlabs.co.uk

19 December 2024 20:10

when you're measuring torch compile warmup "oh 11 secs that's not so bad" then you realize it was 111 secs

Venkatesh Rao 🔹

vgr.bsky.social

19 December 2024 20:09

I’ve had a few like this over the years