Nathan Godey
nthngdy.bsky.social
Post-doc at Cornell Tech NYC Working on the representations of LMs and pretraining methods https://nathangodey.github.io
👤 Joined on 25 November 2024
168
Followers
29
Following
65
Total Posts
📜 Kullanıcı Geçmişi
- 25 Kas 2024, 08:06
🎉 Bluesky'e katıldı - İlk kullanıcı adı:
@nthngdy.bsky.social
🚫 Blokladığı Kullanıcılar
❤️ Beğendiği Gönderiler
Simple, straightforward, and just work well: looping with no KV overhead, and this also gives you a gradients highway that makes backprop work much better
Show Post
A simple mechanism that minimizes overhead in pre-training looped transformers and learns what to store over the recursions through a gradient highway!
Show Post
Very clever! I'd love to see how well this continues to scale out to the monster-sized models!
Show Post
In the memory constrained world we are running into, these kinds of insights are key.
Show Post
⚠️ Stop pretraining your looped Transformer with a separate KV cache per recursion! In our recent preprint, we show that *memory sharing* not only saves memory but is also a net-positive inductive bias! Less memory, same flops, higher quality. 🔗 arxiv.org/abs/2610.02383 🧵
Show Post