One thing I think people confuse with context is they see an LLM has say 400k context and think their codebase is way bigger than that, how can it possibly work. Well, do you hold a 10 million line codebase in your head at once? Of course not. You have an intuitive grasp of how the system is built and laid out, and some general names of things, and before you make a change, you might search through the codebase for specific terms to see what shows up. LLMs do the same thing. They grep through the codebase and read in only files with interesting / matching terms and only the part of the file thats relevant, in much the same way you would open a search result and only view the surrounding method or so. The context is barely used in these scenarios. Context is not something that’s static, it’s built dynamically as the conversation progresses via data coming from your system (partially through tool use).

I frequently use LLMs in a VS Code workspace with around 40 repos, consisting of microservices, frontends, nuget and npm packages, IaC, etc. altogether its many millions of lines of code. and I can ask it questions about anything the codebase and it has no issues managing context. I do not even add files manually to context (this is worse actually because it puts the entire file into context even if it’s not all used). I just refer to the files by name and the LLM is smart enough to read them in as appropriate. I have a couple JSON files that are megs of configuration, and I can tell it to summarize / extract examples out of those files and it’ll just sample sections to get an overview.