The price...

Pretty expensive nowadays, but it’s hard to think that prices won’t come down eventually. AI is being used everywhere, so I’d expect hardware investment to grow exponentially, including in the custom chips that power ultrafast models. To me it feels like we are in the "telephones are for the rich" era of AI history.

Unless we get some kind of very different architecture I’m not sure we’re going to get a 10-100x improvement any time soon, api models will always preference throughput over individual task latency, so any improvement that sacrifices serial decode for better parallelism is going to be more economically viable.

Speed is the real solution here.

I've tried to find a combination of a good-enough model, a VERY fast provider, and cheap enough serving to make this all doable. DeepSeek itself is quite fast, but I crave even more speed (and ideally something hosted outside of China).

Fireworks and Baseten seem to be the leading providers that are available to a solo dev like myself, and support any reasonably decent modern models (DeekSeek 4.x, GLM 5.x, etc). Fireworks has been a real dud, but Baseten in FAST mode has been absolutely amazing. Not cheap - but I find it substantially cheaper than Terra / Sol (low), etc. I haven't found Haiku to be smart enough to be useful.

DeepSeek (the provider) and beyond seem to hit the sweet spot to keep me mostly in-context, so I don't need 4 parallel work items to feel productive. So roughly 200tps or greater. Vanilla Baseten is ~400 tps, and FAST is 700-800 avg.

The other benefit: for my own work in very limited hours, I'm more than happy to pay for answers to come back faster.

speed is also the end of our jobs

Can you explain how please, in detail? I'm failing to see how something that needs a lot of hand holding (the AI we have), when sped up is somehow capable of taking our jobs.

While the LLM speed will increase, some stuff will be incompressible: running unit tests, tool use, UI interaction / mouse control. In my case, a significant amount of the is already spent on that, so much that I rarely ever select fast mode.