I don't think it's generally thought of an advantage that LLMs are stateless. In deployment it's nice, it would be much better if the model could continually learn.

I'm not claiming LLMs are not useful, they most certainly are.