It's not LLMs! Its their providers! If you self-hosted, you'd observe the slow down but you'd get same quality.

Weird! So you're claiming local models somehow are deterministic? Amazing.

All models are deterministic.

There is random input to make the models evaluate differently every time, but without that (or if you fix it) then the model will always give the same output with the same input.

There's no somehow to it, it's actually surprisingly hard to make a computer do something in a non-deterministic fashion.

It's probably not token usage, but a flexible 5 hour limit. There's a reason why they only state it in percentages.

Nope, the issue was not the 5 h limit, but that the 5 hour limit was gone in some minutes and that this burned trough 50% of my weeks fable limit. Before I could do roughly one full 5h session (fable till max) per day.

They're actually extremely predictable. What you are experiencing is the provider you are entrusting with your time, effort, and money fucking you because they need the capacity to train the next one. So without any real notice or indication other than things vaguely seeming broken in ways everyone seems to be experiencing differently, they're serving up some shitty quant of what you already paid to use.

Extremely predictable is an exaggeration. I was teaching machine learning and we needed half a dozen lines to make even simple networks deterministic. LLMs are not really predictable, even if we want to, they will sometimes behave almost chaotic depending on the input.

I'm not sure how you're arriving at "if you use the same seed with the same version of the software on the same hardware with the same drivers you get the same result is an exaggeration", it's wild that you're a teacher and you seemingly do not know this.

Order of operations affects the result in floating point, even moreso at lower precision. GPUs are massively parallel devices with many different sources of latency, so order of operations is not guaranteed for things like dot products, where many threads contribute to the result. Consequently, you can get non-deterministic outputs even though nothing else has changed.

[delayed]

My understanding of LLMs are that they are giant functions and generate output deterministically. However, the way these providers setup and route traffic means that your traffic lands on different boxes where each box has different inputs that run alongside your prompt -- essentially creating "non-deterministic" behavior. Basically it's BS to make LLMs look sentient. Each model could be burned onto an ASIC if you really wanted to be locked into that model and get incredible token speeds. If I'm wrong I would like to be corrected, thanks.

I replied to your neighbor https://news.ycombinator.com/item?id=50432889