ChatGPT makes sense though. They shouldn't be dumping the whole chat into memory the entire time. They're compressing it in some way. If they do it on device, it saves the cost of doing it on the cloud. ChatGPT's memory feature is a good leap ahead of the competition and it could be due to things like this, which may query memories of conversations made long ago and not just the recent ones.

Tools like Cursor do something very similar. They claim to be using 2 million tokens or something, but those are cheap tokens which make the code space more searchable.