Giovanni Monea

giomonea.bsky.social

06 October 2026 16:16

Very excited to finally share my IBM work. It was an intense but rewarding summer, and I had a blast working with my talented collaborators at IBM & Cornell ( @keshavramji.bsky.social @yelkurdi.bsky.social @lastrasl.bsky.social @yoavartzi.com @nthngdy.bsky.social @ramon-astudillo.bsky.social )

Giovanni Monea

giomonea.bsky.social

06 October 2026 16:16

So, what does memory sharing bring to the table to *improve* quality? Prediction recursions attend heavily to the shared memory. Each read sends a gradient signal back to the state recursion. We find that this "gradient highway" makes the state recursion learn much more.

Giovanni Monea

giomonea.bsky.social

06 October 2026 16:16

We train LPT and Hybrid LPT at 150M-1B parameters, on 20-50B tokens at 2, 3, and 5 recursions. The more recursions, the more our memory-sharing models outperform their respective baselines.

Giovanni Monea

giomonea.bsky.social

06 October 2026 16:16

Prior work laid the foundation for looped models and reduced their extra memory at inference. The result? Surprisingly strong despite the train-test mismatch. That made us ask: could this be a good inductive bias? One recursion memorizes, the others reuse it to predict.

Giovanni Monea

giomonea.bsky.social

06 October 2026 16:16

What is memory sharing? Our simple recipe: at later recursions, just concatenate the attention keys of the first recursion and those of the current recursion before performing standard attention. Then, restrict the later recursions to a short window for memory savings.