Yeah, but most LLMs are barely functional after 16k tokens, even if it says 128k on the tin. Sure, they will have recall, but the in-context reasoning ability drops dramatically.

LLMs just aren't smart enough to take over the world. They suck at backtracking, they're pretty bad at world models, they struggle to learn new information, etc. o1, QwQ, and CoT models marginally improve this but if you play with them they still kinda suck