There are simply no models that can keep in context the amount of info required in enterprise codebases before starting to forget or hallucinate.

I've tried to give it relevant context myself (a tedious task in itself to be honest) and even tools that claim to automatically be able to do so fail wonderfully at bigger than toy project size in my experience.

The codebase I'm working on day to day at this moment is give or take around 800,000 lines of code and this isn't even close to our largest codebase since its just one client app for our monolith.

Even trivial changes require touching many files. It would honestly take any average programmer less time to implement something themselves than trying to convince an LLM to complete it.

The largest context that I am aware that an open-source model (e.g. qwen) can manage is 1M tokens. This should translate to ~30kLoC. I'd envision that this could in theory work even on large codebases. It certainly depends on the change to be done but I can imagine that ~30kLoC of context is large enough for most of the module-specific changes. Possibly the models that you're using have a much smaller context window?

Then again, and I am repeating myself from other comments I made here in the topic, there's also Devon which pre-processes the codebase before you can do anything else. That kinda makes me wonder if current limitations that people observe in using those tools are really representative of what might be the current state of the art.

If you don't mind me asking, what size of codebases do you typically work on? As mentioned I've tried using all the available commercial models and none work better than as a helpful autocomplete, test, and utility function generator. I'm sure maybe big players like Meta, OpenAI, MS, etc do have the capability of expanding its context for their own internal projects and training specifically on their code, but most of the rest of us can't feasibly do that since we don't own our own AI moat.

Even on my personal projects and smaller internal projects that are small toy projects or utility tools I sometimes struggle to get them to build anything significant. I'm not saying its impossible, but I always find it best at starting things from scratch, and small tools. Maybe its just a sign that AI would be best for microservices.

I've never used Devon so I can't speak to it, but I do recall seeing it was also overhyped at best and struggled to do anything it was purported to be able to in demos. Not saying that this is still true.

I would be interested in seeing how Devon performs on a large open source project in real-time (since if I recall their demos were not real-time demonstrations) for instance just to evaluate its capabilities.

Several millions lines of code. Can't remember any project that I was involved with and that was less than 5MLoC. C++ system level programming.

Overhyped or not Devon is using something else under the hood since it is pre-processing your whole codebase. It's not "realtime" since it simulates the CoT meaning that it "works" on the patch the very same way a developer would. and therefore it will give you a resulting PR in few hours AFAIR. I agree that a workable example on more complex codebase would be more interesting.

> I've tried using all the available commercial models and none work better than as a helpful autocomplete, test, and utility function generator

That's the why I mentioned qwen because I think commercial AI models do not have such a large window context size. Perhaps, therefore an experience would have been different.

And you have had luck with models like the one you mentioned and Devon generating significant amounts of code in these codebases? I would love to be able to have this due to the productivity gains it should allow but I've just never been able to demonstrate what the big AI coding services claim to be able to do at a large scale.

What they already do is a decent productivity boost but not nearly as much as they claim to be capable of.

As I already said in my first comment, I haven't used those models and any of them would have been forbidden at my work.

My point was rather that you might be observing suboptimal results only because you haven't used the models which are more fit, at least hypothetically, for your use case.

I've heard pretty mixed opinions about the touted capabilities of Devon.

https://www.itpro.com/software/development/the-worlds-first-...

That's good news for us I suppose.

> For example, did you simply ask for the change or did you give a model a chance to learn about the codebase.

I've tried it both with personal projects and work.

My personal project/benchmark is a 3d snake game. O3 is by far the best, but even with a couple of hundred lines of code it wrote itself it loses coherence and can't produce changes that involve changing 2 lines of code in a span of 50 lines of code. It either cannot comprehend it needs to touch multiple places of re-writes huge chunks of code and breaks other functionality.

At work, it's fine for writing unit tests on straight forward tasks that it most likely has seen examples of before. On domain-specific tasks it's not so good and those tasks usually involve multiple file edits in multiple modules.

The denser the logic, the smaller the context where LLMs seem to be coherent. And that's funny, because LLMs seem to deal much better with changing code humans wrote than the code the LLMs wrote themselves.

Which makes me wonder -- if we're all replaced by AI, who will write the frameworks and programming languages themselves?

Thanks but IIUC you're describing a situation where you're simply using a model without giving it a chance to learn from the whole codebase? If so, then I was asking for the opposite where you would ingest the whole codebase and then let the model spit out the code. This in theory should enable the AI model to build a model of your code.

> if we're all replaced by AI, who will write the frameworks and programming languages themselves?

What for? There's enough programming languages and there's enough of the frameworks. How about using an AI model to maintain and develop existing complex codebases? IMHO if AI models become more sophisticated and are able to solve this, then the answer is pretty clear who will be doing it.

Not OP, but I've had the same experience, and that's with tools that purport to handle the context for you.

And frankly, if you can't automate context, then you don't have an AI tool that can realistically replace a programmer. If I have to manually select which of my 10000 files are relevant to a given query, then I still need to be in the loop and will also likely end up doing almost as much work as I would have to just write the code.

I see that you deleted your previous response which was unnecessarily snarky while my question was genuine and simple I suppose.

> And frankly, if you can't automate context,

How about ingesting the whole codebase into the model? I have seen that this is possible with at least one such tool (Devon) and which I believe is using gpt model underneath meaning that other providers could automate this step too. I am curious if that would help in generating more legit large scale changes.

> I see that you deleted your previous response which was unnecessarily snarky while my question was genuine and simple I suppose.

You edited your comment to clarify that you were asking from a place of ignorance as to the tools. Your original comment read as snarky and I responded accordingly, deleting it when I realized that you had changed yours. :)

> How about ingesting the whole codebase into the model? I have seen that this is possible with at least one such tool (Devon) and which I believe is using gpt model underneath meaning that other providers could automate this step too. I am curious if that would help in generating more legit large scale changes.

It doesn't work. Even the models that claim to have really large context windows get very distracted if you don't selectively pick relevant context. That's why I always talk about useful context window instead of just plain context window—the useful context window is much lower and how much you have depends on the type of text you're feeding it.

I don't think my comment read as snarky but I was surprised to see the immediate downvote which presumably came from you so I only added the last sentence. This is a stupid way of disagreeing and attempting to shut down the discussion without merits.

> It doesn't work. Even the models that claim to have really large context windows get very distracted if you don't selectively pick relevant context.

I thought Devon is able to pre-process the whole codebase and which it could take up to a one single day for larger codebases so it must be doing something, e.g. indexing the code? If so, this isn't a context-window specific thing, it's something else and it makes me wonder how that works.

> I don't think my comment read as snarky but I was surprised to see the immediate downvote which presumably came from you so I only added the last sentence.

I can't downvote you because you are downthread of me. HN shadow-disables downvotes on all child and grandchild comments.

I'm the one who upvoted you to counteract the downvote. :)

Ok, sorry about that.

And then they hugged and became lifelong friends :)

You never know - one moment arguing on HN and the second moment you know, drinking at the bar lamenting on how AI is gonna replace us :)

> How about ingesting the whole codebase into the model?

You keep referring to this vague idea of "ingesting the whole codebase". What does this even mean? Are you talking about building a code base specific rag, fine tuning against a model, injecting the entire code base into the system context, etc.?

It is vague because the implementation details you are asking me for are closed source, for obvious reasons. I can only guess what it does but that's besides the point. The point is rather that Devon or 1M window context qwen model might be better or more resilient towards the "lack of context" than what the others were suggesting.