It’s true that you need compute to do large experiments, but the large experiments grow out of small ones. If you can show promising work in a small way, it’s easier to get compute. You can also apply to TRC to get a bunch of TPUs. They had capacity issues for a long time but I’ve heard it’s improved.

Don’t focus on the hype models. Find a niche that you personally like, and do that. If you’re chasing hype you’ll always be skating towards the puck. My original AI interest was to use voice generation to make Dr Kleiner sing about being a modern major general. It went from there, to image gen, to text gen, and kaboom, the whole world blew up. I was the first to show that GPTs can be used for more than just language modeling — in my case, playing chess.

Wacky ideas like that are important to play around with, because they won’t seem so wacky in a year.