Yes × 1000... Jev is the classic type of scientific discovery, any human can see it's effectiveness without actually trying it. We were only deprived the discovery because of the incredible noises created by the AGI/ASI ferver....

I too feel like context management has a lot of room for improvement, but the problem right now is that dynamic context management will most likely tank your cache hit rate. We need caching that's not just based on prefix-matching for this to make sense.

And as I see it that is impossible with current autoregressive global attention models because each part depends on the one before. Maybe if we can cache the local attention in blocks and only recompute the global parts.

My bandaid fix for this is having a large number of sessions or "agents" and being ok with them being inactive at times, but they maintain their context. This has to be in some kind of system with appropriately sized "departments" to justify that agent's existence, or to justify it splitting into two, or creating sub-agents, etc. Yes, there is a cost when you cold start an agent after a week, but for now I'd rather pull that whole 400k+ context back without layers and layers of /compact. And then hopefully you keep that agent going for long enough to benefit from any cache-hit discounts.

Not really a good analogy, but it feels like having an image you like but every time you view it, it runs it through another loop of jpg compression. Use it when it's most effective, otherwise let it preserve it's details.

most people only need if statements

wait till they found out about that nanosecond latency “conditional model”

turns out I’ve been an “RLIF” training expert my whole career

Amusingly the biggest revolution of this tech wave will probably be the automatic if writing.

Completely agree.

It's honestly surprising how little work gets put into building harnesses which carefully steer the LLM through sophisticated context curation. Probably due to the belief that we shouldn't care much because AGI is just around the corner.

Related (but shameless) plug : I've actually been hacking a project over the past two weeks to address this very issue. I'd like to create an SDK which embeds the evaluation of LLMs into code seamlessly, and does "behind the curtains" context curation, to really enable "hybrid" code/LLM agentic applications. (if you're curious => https://github.com/VKCR/semiotique)