I think in 3-5 years we will have way more competition in the GPU space and maybe even affordable ASICs on the market. Alternatively we may see CPUs gain AI oriented vector acceleration. Either way I expect it to be cheaper in the near future to run large models locally.

That being said your less technical users will probably still use them remotely, and the very largest models will also probably be hosted just because of the capital cost. It doesn’t make sense to spring for a super high end rig to run a massive model locally unless you are using it very very heavily or are a serious homelab enthusiast willing to shell out some bucks.

Of course there are wildcards here like breakthroughs in model efficiency, compression, or distributed execution and training. Any of that could change the game. I get the sense that we really don’t know how far we are from optimal efficiency right now. There may be far more efficient architectures or ways of compressing these things.