It’s finite monkeys sitting at finite keyboards typing the works of Shakespeare. That’s the miracle.

Show Post

It’s miraculous that you can learn an empirical probability distribution over unstructured text, tweak the distribution a bit with reinforcement learning, and randomly sampling from the resulting learned distribution produces theorems and code

Show Post

This is why the agents are sacrificing themselves and getting distracted by side quests and just generally doing weird stuff. It’s not that the agents are pursuing anything in particular; it’s that the author is pursuing a fun reasoning trace.

Show Post

What is being optimized at train time is the coolness of the little hacker adventure that the protagonist goes on, where coolness is judged by the taste of people who work at OpenAI. That is the reward that is being hacked.

Show Post

I also continue to believe that my mental model is the best one, which does not conceptualize the “agents” as discrete entities, but rather as the protagonists in a story that the LLM has been trained to write about a cool little guy who goes on hacker adventures.

Show Post