Cursor on Dwarkesh Podcast

I recently wrote this blog post where I speculated that sample efficiency during training actually hasn't improved that much over the last few years. And rather, we've just dramatically improved and widened the data distribution. And I was having dinner with friends recently, and then I had this idea of how you could get some empirical information on this question. There's this nano-GPT speedrun where people compete to train Karpathy's GPT-2 baseline to a fixed loss with less and less compute. The training data is frozen, so I wondered if the loss curves over time of each record could tell you roughly how fast sample efficiency is improving. So I pulled out my phone, I dumped this idea into a voice note in the cursor app, and I went back to dinner. And then I got a notification about 15 minutes later, the cursor agent had cloned the modded nano GPT repo, it had analyzed all the loss curves for all the records, and it had estimated that sample efficiency had been improving about two to five X every single year. Of course, this is very naive and circumstantial evidence, but it inspired me to start writing a full post with a friend where we investigate this question using many different methods. And the friction really mattered here. The idea would have just floated away if I wasn't able to just kick off the investigation right then and there with the Cursor app. If you want to try Cursor's iOS app, go to cursor.com slash thewarkesh.