Keshav Ramji
@keshavramji.bsky.social did:plc:d5cez6jshyuvsazkw7fg2oyn
working toward continually self-improving AI, reasoning and alignment @ IBM Research AI | Prev: Penn CS + Wharton
Sur Bluesky depuis le
- Abonnés
- 972
- Abonnements
- 228
- Posts
- 14
- Jours dans les classements
- 0
Résumé
Keshav Ramji (@keshavramji.bsky.social) est sur Bluesky depuis le 13 novembre 2024. Le compte a 972 abonnés, suit 228 comptes et a publié 14 posts.
Présence dans les classements
Les meilleurs jours où les posts de ce compte sont entrés dans le top quotidien de Debya.
Ce compte n’est encore apparu dans aucun classement quotidien.
Posts aimés récemment
J’aime publics du réseau Bluesky.
Very excited to finally share my IBM work. It was an intense but rewarding summer, and I had a blast working with my talented collaborators at IBM & Cornell ( @keshavramji.bsky.social @yelkurdi.bsky.social @lastrasl.bsky.social @yoavartzi.com @nthngdy.bsky.social @ramon-astudillo.bsky.social )
Voir le postSo, what does memory sharing bring to the table to *improve* quality? Prediction recursions attend heavily to the shared memory. Each read sends a gradient signal back to the state recursion. We find that this "gradient highway" makes the state recursion learn much more.
Voir le postWe train LPT and Hybrid LPT at 150M-1B parameters, on 20-50B tokens at 2, 3, and 5 recursions. The more recursions, the more our memory-sharing models outperform their respective baselines.
Voir le postPrior work laid the foundation for looped models and reduced their extra memory at inference. The result? Surprisingly strong despite the train-test mismatch. That made us ask: could this be a good inductive bias? One recursion memorizes, the others reuse it to predict.
Voir le postWhat is memory sharing? Our simple recipe: at later recursions, just concatenate the attention keys of the first recursion and those of the current recursion before performing standard attention. Then, restrict the later recursions to a short window for memory savings.
Voir le post
Comptes bloqués
Sur Bluesky, les blocages sont publics.
Ce compte ne bloque personne.