I'm a bit nervous where I'd end up though - with code I'd "written" but wasn't familiar with, and with who knows what kinds of limitations or subtle bugs baked in.
This does seem like quite a big downside. It turns every new feature into “implement this in someone else’s code base”. I imagine you’d very quickly have complete dependency on the AI. Maybe that’s an inevitability in this new world?
That's a feature, not a bug. At least from OAI/Anthropic's point of view.
I don't think there's any current AI that is fully trustworthy this way though.
I wouldn't even put them at 50% trustworthy
I think we are going to see a cliff where they become 80% good, and every tiny bit of improvement past that point will be exponentially more difficult and expensive to achieve. I don't think we reach 100% reliable AI in any of our lifetimes
Current AI is likely already beyond 50% trustworthiness, whatever that means.
People had this same prediction about offshore development
Those old school devs are able to find well paying work fixing broken software churned out by overseas code sweatshops
I predict if you can read and understand code without the help of AI models you will be in even higher demand to fix the endless broken software built by AI assisted coders who cannot function without AI help
It amazes me that more people aren't worried about this.
I’m having a hard time coming close to the 500 included in the monthly subscription and I use it like, a lot.
Just curious how you’re hitting that 200-300 mark unless you’re talking about paying Anthropic outside of cursor. Which I just now realized is probably the case.
Also (most) engineers don't hallucinate answers. Claude still does regularly. When it does it in chat mode via a flat rate Pro plan I can laugh it off and modify the prompt to give it the context it clearly didn't understand but if its costing me very real money for the LLM to over-eagerly over-engineer an incorrect implementation of the stated feature its a lot less funny.
Right now with Copilot or other fixed subscriptions I can also laugh it off and just create a new tab with fresh context. Or if I get rate-limited because of too much token use I can wait 1 day. But if these actions are linked to directly costing money on my card, then that's becoming a lot more scary.
They absolutely do, where do you think bugs come from? The base rate is typically just lower than current AIs.
For exemple I can’t remember seing a PR with an API that seems plausible but never ever existed, or an interpretation of the specs so convoluted and edgy that you couldn’t even use sarcasm as a justification for that code.
Don’t take me wrong: some LLMs are capable of producing bugs that looks like humans ones, but the term hallucinate is something else’s and doesn’t fit with much humans bugs.
A PR is code that has already been tested and refined, which is not comparable to the output of an LLM. The output of an LLM is comparable to the first, untested code that you wrote based off of your sometimes vague memory of how some API works. It's not at all uncommon to forget some details of how an API works, what calls it supports, the details of the parameters, etc.
It won't say "I think it should look something like this but I might be wrong," it'll say "simple! here is how you do it."
Hence hallucination and not error. It thinks it's right.
For an LLM-equivalent experience, go open notepad.exe and make substantial changes there, and then rebuild - and let the compiler tell you what's your base rate of hallucinating function names and such.
And we never made up APIs, as there just weren't that many APIs. We would open the .h file for the API we were targeting as we typed into the other window. And the LLMs have ingested all the documentation and .h files (or the modern equivalent) so they don't have a real excuse.
But I use the LLMs all the time for math, and they do profusely hallucinate in a way people do not. I think it's a bit disingenuous to say that LLMs don't have that failure mode that people don't really have.
And it doesn't fight me like the OpenAI tooling does that logs me out randomly every day and I have to login and spend 4 minutes copying login codes from my email or answering their stupid Captcha test. And this is on their API playground where I pay for every single call - so not like I'm trying to scrape my free chat usage as an API.
Okay maybe need to clarify: I hit those limits when I do agentic stuff, which is what Claude Code does: So let the LLM automatically pull in files into the context it thinks it needs, analyze my codebase, follow imports, add more code, etc. It can quickly balloon out of control when the LLM pulls in too many LoC and the context window gets too big.
Then do a few back and forth actions like "let's refine this plan, instead of X pls do Y", or "hmm I think maybe we should also look into file blah.ts" and you quickly hit 500k tokens.
If I use Cody only, which has some agentic capabilities but is much more "how can I implement Y in this file @src/file1.ts db models are in @src/models/foo.ts", then I rarely ever hit any rate limitations. That's more similar to what you describe of copying code back and forth, except it's in the editor and you can do it by writing @somefile.
A small change to create a script file (20 LoC) was 10cts, a quick edit to a README was 7ct
Yes yes engineers make more than that blah blah but the cost would quickly jump out of control for bigger tasks. I’d easy burn through $10-20 upwards a day with this, or upwards $100-$300 a month. Unless you have a Silicon Valley salary, that’s too expensive.
I use other tools like Cody (the tool the author created) or Copilot because I pay $10 a month and that’s it. Yes I get rate limited almost daily but I don’t need to worry that my tool cost is going out of control suddenly.
I hope Anthropic introduces a new plan that bundles Claude Code into it, I’d be much more comfortable using that knowing it won’t suddenly be more than my $50/mo (or whatever)