Why 25% and not 12% or 40%? Is this an arbitrary vibestimate or you had some tests done that pointed you to 25%? I am genuinely interested in how others deal with context issues. Also I would think the usable context window is variable depending on the task, for example summarizing documents vs analyzing large, scattered and complex instructions.

In my own experience (both Claude and Gemini 1M), you start running into problems around 60-70%. A 25% ceiling sounds like total cargo culting to me.

Benchmarks show a large drop in quality as context grows. Your opus will be acting like haiku above 300k tokens.

I'm not the above poster, but my experiences generally line up with theirs.

The first 20% of context is usually very solid. Agent stays on task, is responsive and focused on the original goal.

The last 20% of context is usually pretty bad. Agent tends to loop, repeat, drift away into sub-goals (or non-goals).

The middle 60% is a bit of a grab-bag. Sometimes it's fine, sometimes it's not. It varies by task, prompt, and luck.

The above 25% and 50% limits seem like reasonable and simple rules to keep agents in the productive zones of usage.

It's generally not worth doing extensive testing to try to find your exact threshold unless you're running exactly the same prompt at scale, on similar data.

If you are in that spot, I won't actually recommend langchain tooling (I've been somewhat frustrated with it) but I think the concepts are worth understanding.

You can do something like https://docs.langchain.com/oss/python/langchain/test/evals. Trajectory evals in particular seem to suffer as context length grows (https://docs.langchain.com/oss/python/langchain/test/evals#t...).

I've used this for testing an agent that handles provisioning and permissions for developers with SaaS tooling (ex - if a developer asks, go provision an account [saml/oauth/etc] with the 3rd party, configure access, and let them know how to use it).

I'm about 6 months out of date at this point, so I won't speak for the latest models, but what was available early this year absolutely suffers from extended context lengths.

250k because thats what the smaller models supported and theres better training data in that part of the window

there have been some papers suggesting that the useful context is even smaller, and stays fixed as you change the context window size.

as a more general case though, i think the possibilities for what youd need to include in training to have many different paths of text be well represented enough over the long window means the later tokens will almost always be a lot more random than early ones?

there's a sheer amount of bits problem.

>in that part of the window

What matters is the relative distance. Using the latter 250k should be equivalent to using the first 250k.

Every single analysis I've seen done by anyone has basically come to the same conclusion: assuming sane BF16 or Q8 model quant, and BF16 or Q8 KV context quant (ie, not intentionally screwing over the model), you get about 250k before it pukes.