It's pretty painful to have speeds < 30 tok/sec though. Especially if you're used to API providers at higher speeds. It makes any interactive work almost impossible to do efficiently because you have no choice but to context switch after every request.

I assume it will get better over time, and does it improve in speed if you use a larger buffer? Say instead of 2GB you go with 6GB? I imagine it would, and you might need to stream drastically less no?

Yeah, correct! You can set up this engine to get more expert cache slots (e.g 32 instead of 16) to get a better hit rate and better tok/s. it will be 3.5gb instead of 2gb.