It’s true that you need compute to do large experiments, but the large experiments grow out of small ones. If you can show promising work in a small way, it’s easier to get compute. You can also apply to TRC to get a bunch of TPUs. They had capacity issues for a long time but I’ve heard it’s improved.

Don’t focus on the hype models. Find a niche that you personally like, and do that. If you’re chasing hype you’ll always be skating towards the puck. My original AI interest was to use voice generation to make Dr Kleiner sing about being a modern major general. It went from there, to image gen, to text gen, and kaboom, the whole world blew up. I was the first to show that GPTs can be used for more than just language modeling — in my case, playing chess.

Wacky ideas like that are important to play around with, because they won’t seem so wacky in a year.

Thats interesting, at a glance the TRC thing looks more altruistic and impactful than what I had in mind for learning or making money. I'll have to keep it in mind if I do, do something share worthy ever. Thanks!

Another area where there's potential for an individual to make lots of progress is in theory and mechanistic interpretation. Although it's not where the money is, it's probably not rapid progress and it's really hard.

There's API access for GPUs you can rent, as well as model specific APIs like for Stable Diffusion or GPT 4. You can do a lot as a solopreneur now.

For example, Tony Dinh made a macOS GPT wrapper and makes like 40k a month from it, just utilizing OpenAI's APIs: https://news.ycombinator.com/item?id=37622702

I'm not in ML - here a quick take

In a gold rush, sell shovels! The ML pipeline has a lot of bottlenecks. Work on one, get useful and novel expertise, and have a massive impact on the industry. Like maybe you could find a way to optimise your GPU usage? Is there a way to package what you feed it more efficiently?

The point not being of competing with OpenAI, but to solve a problem that everyone in the field has.