How I am tackling this problem is making it dead simple for users to create analyzers that are designed to enriched text data. You can read more about how it would be used in a search at https://github.com/gitsense/chat/blob/main/packages/chat/wid...
The basic idea is, users would construct analyzers with the help of LLMs to extract the proper metadata that can be semantically searched. So when the user does an AI Assisted search with my tool, I would load all the analyzers (description and schema) into the system prompt and the LLM can determine which analyzers can be used to answer the question.
A very simplistic analyzer would be to make it easy to identify backend and frontend code so you can just use the command `!ask find all frontend files` and the LLM will construct a deterministic search that knows to match for frontend files.
I suppose the problem you have might be unique to nextJS ?
Now imagine as part of your analyzer, you have the following instructions for the llm:
--- For all files in `src/frontend/` treat them as frontend code. For all files in 'src/' excluding `src/frontend` treat as backend. Create a metadata called `scope` which can be 'frontend', 'backend' or 'mix' where mix means the code can be used for both front and backend like utilities.
Now for each file, create a `keywords` metadata that includes up to 10 unique keywords that describes the core functionality for the file. ---
So with this you can say
- `!ask find all frontend files`
- `!ask find all mix use files`
- `!ask find all frontend files that does [this]`
and so forth.
The whole point of analyzers is to make it easy for the LLM to map your natural language query to a deterministic search.
If the code base is straightforward and follows a well known framework, asking for frontend or backend wouldn't even need an entry as you can just include in the instructions that I use framework X and the LLM would know what to consider.
I don't believe any human can understand a problem if they need to fit the entire problem blem domain in their head, and the scope of a domain that doesn't fit on a computer. You have to break it down into a manageable amount of information to tackle it in chunks.
If a person can do that, so can an LLM prompted to do that by a person.
My ability to break a problem down does not start from listing the files out and reading a few. I have a high level understanding of the whole project at all times, and a deep understanding of the whole project stored, and I can recall that when required, this is not true of an LLM at any point.
We know this is a limitation and it's why we have various tools attempting to approximate memory and augment training on the fly, but they are approximations and they are in my opinion, not even close to real human memory and depth of understanding for data it was not trained on.
Even for mutations of scenarios it was trained on, which code is a great example of that. It is trained on billions of lines of code, yet still fails to understand my codebase intuitively. I have definitely not read billions of lines of code.
The bigger the context, the more stuff "decays" sometimes to complete different meanings
The important thing in this context is that I know it's all there, I don't have to grep the codebase to fill up my context, and my understanding of the holistic project does not change each time I am booted up.
Such as ctrl-shift-f
But this is an advanced topic, I don't wanna get into it
All 3 points (you have had all of it your head at some point, it is still there, that is not true of an LLM) are mere conjectures, and not provable at this time, certainly not in the general case. You may be able to show this of some codebases for some developers and for some LLMs, but not all.
Even remembering alters the memory being recalled, entirely unlike how computers work.
You may want to stay home then to avoid getting lost.
I think it's objectively true that the information is not in the LLM. It did not have all codebases to train with, and they do not (immediately) retrain on the codebases they encounter through usage.
I does, it’s just happening at lightning speed.
If we had that level of understanding of how exactly our brains do what they do things would be quite different.
If you're completely new to the problem then ... yes, it does.
You're assuming that you're working on a project that you've spent time on and learned the domain for, and then you're comparing that to an LLM being prompted to look at a codebase with the context of the files. Those things are not the same though.
A closer analogy to LLMs would be prompting it for questions when it has access (either through MCP or training) to the project's git history, documentation, notes, issue tracker, etc. When that sort of thing is commonplace, and LLMs have the context window size to take advantage of all that information, I suspect we'll be surprised how good they are even given the results we get today.
Of course, because I am not new to the problem, whereas an LLM is new to it every new prompt. I am not really trying to find a fair comparison because I believe humans have an unfair advantage in this instance, and am trying to make that point, rather than compare like for like abilities. I think we'll find even with all the context clues from MCPs and history etc. they might still fail to have the insight to recall the right data into the context, but that's just a feeling I have from working with Claude Code for a while. Because I instruct it to do those things, like look through git log, check the documentation etc, and it sometimes finds a path through to an insight but it's just as likely to get lost.
I alluded to it somewhere else but my experience with massive context windows so far has just been that it distracts the LLM. We are usually guiding it down a path with each new prompt and have a specific subset of information to give it, and so pumping the context full of unrelated code at the start seems to derail it from that path. That's anecdotal, though I encourage you to try messing around with it.
As always, there's a good chance I will eat my hat some day.
That is true for the LLMs you have access to now. Now imagine if the LLM had been trained on your entire code base. And not just the code, but the entire commit history, commit messages and also all of your external design docs. And code and docs from all relevant projects. That LLM would not be new to the problem every prompt. Basically, imagine that you fine-tuned an LLM for your specific project. You will eventually have access to such an LLM.
Conceptually, there is no difference between fine-tuning the LLM for being a law expert of specific country and fine-tuning the LLM for being an expert for given codebase. Former is already happening and is public. Latter is not yet public but I believe it is happening.
Reason why big co are pursuing generic LLMs is because they serve as a foundation for basically any other derivative and domain-specific work.
Consider that you're coding a Linux device driver. If you ask for help from an LLM that has never seen the Linux kernel code, has never seen a Linux device driver and has never seen all of the documentation from the Linux kernel, you're going to need to provide all of this as context. And that's both going to be onerous on you, and it might not be feasible. But if the LLM has already seen all of that during training, you don't need to provide it as context. Your context may be as simple as "I am coding a Linux device driver" and show it some of your code.
I don’t see any progress on incrementally training LLMs on specific projects. I believe it’s called fine tuning, right?
Why isn’t that the default approach anywhere instead of the hack of bigger “context windows”?
Also, you're a programmer you have no foundation of knowledge on which to make that assessment. You might as well opine on quarks or martian cellular life. My god the arrogance of people in my industry.
I thought it was rare.
I don't believe these people are lying. They are self-reporting their own experiences, which unfortunately have the annoying property of being generated by the very mind that is living the experience.
What does it mean to have an eidetic memory? It means that when you remember something you vividly remember details, and can examine those details to your heart's content. When you do so, it feels like all those details are correct. (Or so I'm told, I'm the opposite with aphantasia.)
But it turns out if you actually have a photo reference and do a blind comparison test, people who report photographic memories actually don't do statistically any better than others in remembering specific fine details, even though they claim that they clearly remember.
The simpler explanation is that while all of our brains are provide hallucinated detail to fill the gaps of memories, their brains are wired up to present those made up details feel much more real than they do to others. That is all.
This may change your mind!
There are so many well documented cases of idiot savants with insane memory skills in various areas (books, music, dates/events, etc), that this type of snapshot visual memory (whatever you want to call it) doesn't seem surprising in that context - it'd really be a bit odd such diverse memory skills excluded one sensory modality (and it seems it doesn't).
Hearsay is not reliable. Yes there are stories of savants. When you put them in a lab and actually see how good their memory is, it turns out to be roughly the same as everyone else's. The stories aren't true.
(They may be better at remembering weird facts or something, but when you actually calculate the information entropy of what they are remembering, it ends up being within the ballpark of what a neurotypical person remembers across of general span of life. That's why these people are idiot savants (to use your term). They allocate all their memory points to weird trivia and none to everyday common knowledge.
I think it's more complex than that - it's they way they are forming memories (i.e. what they remember) that is different to a normal person. In a normal person surprise/novelty (prediction failure) is the major learning signal that causes us to remember something - we're selective in what gets remembered (this is just mechanically how a normally operating brain works), whereas the savant appears to remember everything in certain modalities.
I don't think that "using up all their memory" is why savants are "idiots", but rather just a reflection of something more severe that is wrong.
Engineering is merely a search for optimal solution in this multidimensional space of problem domain(-s), requirements, limitations and optimization functions.
The best engineers understand how big a difference that is
How often do you need more than 10 million tokens to answer your query?
I haven't used the Llama 4 10 million context window so I don't know how it performs in practice compared to the major non-open-source offerings that have smaller context windows.
But there is an induced demand effect where as the context window increases it opens up more possibilities, and those possibilities can get bottlenecked on requiring an even bigger context window size.
For example, consider the idea of storing all Hollywood films on your computer. In the 1980s this was impossible. If you store them in DVD or Bluray quality you could probably do it in a few terabytes. If you store them in full quality you may be talking about petabytes.
We recently struggled to get a full file into a context window. Now a lot of people feel a bit like "just take the whole repo, it's only a few MB".
Maybe models would get better in picking up relevant information from large context, but AFAIK it is not the case today.
The attention mechanism that transformers use to find information in the context is, in its simplest form, O(n^2); for each token position, the model considers whether relevant information has been produced at the position of every other token.
To preserve performance when really long contexts are used, current-generation LLMs use various ways to consider fewer positions in the context; for example, they might only consider the 4096 "most likely" places to matter (de-emphasizing large numbers of "subtle hints" that something isn't correct), or they might have some way of combining multiple tokens worth of information into a single value (losing some fine detail).
Thanks but I don't and I'm not sure why you're jumping to this conclusion.
Selectively gripping and searching the code to pull into context only those parts relevant to the task at hand.
And that context is the valuable bit for quickly getting back up to speed on a codebase.
Agreed but that’s a bit different from “the context is the repo”
It’s been my experience that usually just picking a couple files out to add to the context is enough - Claude seems capable of following imports and finding what it needs, in most cases.
I’m sure it depends on the task, and the structure of the codebase.
No it’s in the problem at hand. I need to load all related files, documentation, and style guides into the context. This works really well for smaller modules, but currently falls apart after a certain size.
The context is in the repo. An LLM will never have the context you need to solve all problems. Large enough repos don't fit on a single machine.
There's a tradeoff just like in humans where getting a specific task done requires removing distractions. A context window that contains everything makes focus harder.
For a long time context windows were too small, and they probably still are. But they have to get better at understanding the repo by asking the right questions.