Michael Noukhovitch

mnoukhov.bsky.social

17 September 2026 05:45

Is RL actually making your LLM better? Gains from RL are mostly on easy questions🤯 We're calling this the Matthew Effect for RL on LLMs. We then leverage async RL to solve harder problems by Never Giving Up! arxiv.org/abs/2609.13443 and mnoukhov.github.io/posts/ngu/ and check out thread below 🧵👇

Michael Noukhovitch

mnoukhov.bsky.social

17 September 2026 05:45

We look at the initial, pre-RL model's accuracy on each eval example. Then track how it changes over time. The majority of gains from RL are coming from easy problems going from mostly solved to fully solved i.e. "the rich get richer" aka the Matthew Effect.

Michael Noukhovitch

mnoukhov.bsky.social

17 September 2026 05:45

Why is this happening? Maybe we just need to take more completions k for each prompt when doing GRPO. That actually makes things worse. K=4 beats K=32 compute-matched on the hardest subset of questions

Michael Noukhovitch

mnoukhov.bsky.social

17 September 2026 05:45

Want to spend more compute on hard questions? Instead just spend less compute on easy questions! Smaller k=4 filters more easy prompts which reallocates compute to hard questions in async RL

Michael Noukhovitch

mnoukhov.bsky.social

17 September 2026 05:45

To combine the best of big k and small k, we can just take 4 samples and if they're all wrong...Never Give Up! Sample 4 more completions with probability p. NGU solve hard prompts like k=32 and filters easy prompts like k=4.