I do believe it is a language model, and not a repository of pure fact, but it remembers context, because it is designed for conversation. It is augmented to remember things that you have talked about, thus, your current talking points/interests.

For example: you talk about philosophers. Then you refer to them as "they". From then on, it will remember the context whenever you refer to philosophers as "they". When pressed, it will repeat your mantra of "I am just a language model, I can't possibly remember context" while it clearly does, either by design or emergence.

That's only because you don't give it just a single prompt like you would with a human during conversation. When you add a new sentence, what you're actually doing is giving it the whole history of your conversation with your new sentence appended at the end, so it always analyzes the whole before returning a new answer. It doesn't remember context between prompts, it's only the UI that makes you think it does.

It's good to think about these models as eager cosplayers of any role you set up in your prompt. In this case, it sees its own past replies, so it cosplays an AI that gave these answers before. You could edit its past replies and override its "personality" this way.

Oh okay, I suppose re-evaluating the entire conversation history for every response would make it stateless.