Claude Sonnet 4 now supports 1M tokens of context

Re:Claude Sonnet 4 now supports 1M tokens of context by brulard on Tuesday August 12, 2025 @08:27PM (#44883346)

I think you misunderstand how context in current LLMs works. To get the best results you have to be very careful to provide what is needed for immediate task progression, and postpone context thats needed later in the process. If you give all the context at once, you will likely get quite degraded output quality. Thats like if you want to give a junior developer his first task, you likely won't teach him every corner of your app. You would give him context he needs. It is similar with these models. Those that provided 1M or 2M of context (Gemini etc.) were getting less and less useful after cca 200k tokens in the context.

Maybe models would get better in picking up relevant information from large context, but AFAIK it is not the case today.

[ Parent | on HN ]

Re:Claude Sonnet 4 now supports 1M tokens of context by remexre on Tuesday August 12, 2025 @10:31PM (#44884114)

That's a really anthropomorphizing description; a more mechanical one might be,

The attention mechanism that transformers use to find information in the context is, in its simplest form, O(n^2); for each token position, the model considers whether relevant information has been produced at the position of every other token.

To preserve performance when really long contexts are used, current-generation LLMs use various ways to consider fewer positions in the context; for example, they might only consider the 4096 "most likely" places to matter (de-emphasizing large numbers of "subtle hints" that something isn't correct), or they might have some way of combining multiple tokens worth of information into a single value (losing some fine detail).

[ Parent | on HN ]

Re:Claude Sonnet 4 now supports 1M tokens of context by ants_everywhere on Wednesday August 13, 2025 @08:15AM (#44887497)

> I think you misunderstand how context in current LLMs works.

Thanks but I don't and I'm not sure why you're jumping to this conclusion.

[ Parent | on HN ]

Re:Claude Sonnet 4 now supports 1M tokens of context by brulard on Wednesday August 13, 2025 @03:47PM (#44892943)

I said that based on you saying you exhaust a million token context windows easily. I'm no expert on that, but I think the current state of LLMs works best if you are not approaching that 1M token limit, because large context (reportedly) deteriorates response quality quickly. I think state of the art usage is managing context in tens or low hundreds thousands tokens at most and taking advantage of splitting tasks across subtasks in time, or splitting context across multiple "expert" agents (see sub-agents in claude code).

[ Parent | on HN ]

Re:Claude Sonnet 4 now supports 1M tokens of context by jimbokun on Wednesday August 13, 2025 @12:32PM (#44890614)

It seems like LLM need to become experts at managing their OWN context.

Selectively gripping and searching the code to pull into context only those parts relevant to the task at hand.

[ Parent | on HN ]

Re:Claude Sonnet 4 now supports 1M tokens of context by brulard on Wednesday August 13, 2025 @03:38PM (#44892840)

That's what I'm thinking about a lot. Something like the models "activate" just some subset of parameters when working (if I understand the new models correctly). So that model could activate parts of context which are relevant for the task at hand

[ Parent | on HN ]