Fair enough.

I'd seriously recommend the 4090 over the 4070 if you want to do finetuning/inference locally. And I highly recommend 64GB of ram.

The 24GB of VRAM is 100% worth it alone. If you want to do local ML stuff you _need_ that 24gb of VRAM.

64GB of ram + 24GB of vram lets you run a lot of the medium size models at decent speeds. I don't use Colab personally but AFAIK it should work fine for you if you don't want to do it locally.

Also worth noting is the newer ray tracing rendering that cyberpunk is doing. You should checkout the demos IMO it looks sick. It only runs at 18fps on a 4090 so it's only playable on a 4090 + dlss, and I'm not sure if the newer rending tech will be super achievable on any of the other cards - if that's of interest to you.

Get a 13900 or 7950X and 64 gb of ram. You can run llama 30B and 65B, slowly but surely. Play with that before buying a gpu. If you really, really see yourself getting into this, then go ahead and get a 3090 or 4090. But otherwise get a cheaper nvidia card and wait for things to develop a little more. You can still play with ML and CUDA but you'll have cash left for when 50X0s drop, and that will probably be right around when this stuff will really be getting hot (if the current plateau doesn't hold).

Llama is basically an auto-complete right now. We're celebrating baby's first steps. It's not really worth the $600-1000 jump up from cards that can run all current games 4k60.