News for hackers. Stuff that matters.
Re:The great AI delusion is falling apart by simonw on Thursday July 24, 2025 @01:22PM (#44673411)
That METR paper - discussed here two weeks ago: https://news.ycombinator.com/item?id=44522772 - has so much sticking power.

It's not a bad paper, but it's also turning into a fantastic illustration of how much thirst there is out there for anything that shows that AI productivity doesn't work.

I just learned there was a 4 minute TV news segment about it on CNBC! https://www.youtube.com/watch?v=WP4Ird7jZoA

Re:The great AI delusion is falling apart by timr on Thursday July 24, 2025 @01:24PM (#44673438)
> It's not a bad paper, but it's also turning into a fantastic illustration of how much thirst there is out there for anything that shows that AI productivity doesn't work.

Maybe. I think it's a fantastic illustration of anyone doing anything to provide something other than hype around the subject. An actual RCT? Too good to be true! The thirst is for fact vs. speculation, influencer blogposts and self-promotion.

That this RCT provided evidence opposing the hype, is, of course, irresistible.

Re:The great AI delusion is falling apart by simonw on Thursday July 24, 2025 @01:37PM (#44673627)
That's what I liked about the paper: the methodology felt better to me than many other studies. I noted that when I blogged about it (second-to-last paragraph https://simonwillison.net/2025/Jul/12/ai-open-source-product... )
Re:The great AI delusion is falling apart by pydry on Thursday July 24, 2025 @03:51PM (#44675235)
The other papers ive seen so far all used toy projects. This paper was the first ive seen that deliberately used realistic projects.

I think that part is understated, while the part about the "steep learning curve" has been considerably overstated. Ive watched plenty of good developers with hours of cursor practice, using plenty of relevant context giving clear instructions wasting chunks of 5-10 minutes at a time because 4/5 times claude produced crap that needed to be undone.

That story doesnt sell though. Nobody will pay me consulting fees to say it. Hype has more $$$ attached.

Re:The great AI delusion is falling apart by ofjcihen on Thursday July 24, 2025 @01:34PM (#44673577)
To be fair that thirst probably comes from people who aren’t seeing the gains the hype would lead you to believe and are reaching into the void to not feel like they’re taking crazy pills.

It’s also probably not coming from a place of “I’m scared of AI so I want it to fail” but more like “my complex use case doesn’t work with AI and I’m really wondering why that is”.

There’s this desire it seems to think of people who aren’t on the hype train as “against” AI but people need to remember that these are most likely devs with a decade of experience who have been evaluating the usefulness of the tools they use for a long time.

Re:The great AI delusion is falling apart by jsbisviewtiful on Thursday July 24, 2025 @02:02PM (#44673940)
Personal take, but I think some people also understand that the hype machine around AI is coming from the rich and C-level people. Meanwhile companies are widely and openly axing jobs and or not paying artists, citing AI as the source of their new fortune. Personally, my use of AI in my job has so far not been that fruitful and for something that has so far dramatically underdelivered on its promises of utopian ideas we are instead actively seeing it used to undercut the 99% - and that’s not even getting into the environmental impact or the hellscape it’s made of the Internet.
Re:The great AI delusion is falling apart by archagon on Thursday July 24, 2025 @04:28PM (#44675689)
It’s telling that there seem to be no AI companies with “boring” CEOs. It’s all glassy-eyed zealots talking about how their product will upend the workforce.
Re:The great AI delusion is falling apart by Terr_ on Thursday July 24, 2025 @02:03PM (#44673961)
Trying out an analogy:

The time is the early 2000s, and the Segway™ is being suggested as the archetype of almost all future personal transportation in cities and suburbs. I don't hate the product, there's neat technology there, they're fun to mess with, but... My bullshit sensor is still going off.

I become tired of being told that I'm just not using enough imagination, or that I would understand if only I was plugged into the correct social-groups of visionaries who've given arguments I already don't find compelling.

Then when somebody does a proper analysis of start/stop distance, road throughput, cargo capacity, etc, that's awesome! Finally, some glimmer of knowledge to push back the fog of speculation.

Sure, there's a nonzero amount of confirmation bias going on, but goshdangit at least I'm getting mine from studies with math, rather than the folks getting it from artistic renderings of self-balancing vehicles filling a street in the year 2025.

Re:The great AI delusion is falling apart by Yoric on Thursday July 24, 2025 @05:27PM (#44676418)
> To be fair that thirst probably comes from people who aren’t seeing the gains the hype would lead you to believe and are reaching into the void to not feel like they’re taking crazy pills.

Yes, exactly!

I've spent way too much time trying to get anything remotely close to an LLM writing useful code. Yeah, I'm sure it can speed up writing code that I can write in my sleep, but I want it to write code I can learn from, and so far, my success rate is ~0 (although the documentation along the bogus code is sometimes a good starting point).

Having my timelines filled by people who basically claim that I'm just an idiot for failing to achieve that? Yeah, it's craze-inducing.

Every time I see research that appears to confirm the hype, I see a huge hole in the protocol.

Now finally, some research confirming my observations? It feels so good!

Re:The great AI delusion is falling apart by goalieca on Friday July 25, 2025 @11:38AM (#44684401)
For some, it is also coming from a place that their company leadership is mandating AI use.
Re:The great AI delusion is falling apart by emp17344 on Thursday July 24, 2025 @01:33PM (#44673564)
Well, there aren’t any studies showing AI agents boost productivity, so it’s all we’ve got. It seems like a well-conducted study, so I’m inclined to trust its conclusions.
Re:The great AI delusion is falling apart by simonw on Thursday July 24, 2025 @01:40PM (#44673664)
One of the articles linked from the OP includes links to such studies: https://theconversation.com/does-ai-actually-boost-productiv... - scroll down to the "AI and individual productivity" section, there are two papers there on the "increases productivity" side followed by two others that didn't.
Re:The great AI delusion is falling apart by bluefirebrand on Thursday July 24, 2025 @01:26PM (#44673472)
Anyone who is an employee drawing a salary should be extremely hopeful that AI productivity doesn't work.

Why should we be eager to find out that some new tech is going to undercut us and replace us, devaluing us even more than we already are?

Re:The great AI delusion is falling apart by simonw on Thursday July 24, 2025 @01:41PM (#44673678)
Do you benefit from open source?

Open source packages are the biggest productivity boost of my entire career, at no point did I think "wow, I wish these didn't exist, they're a threat to my livelihood".

Re:The great AI delusion is falling apart by bluefirebrand on Thursday July 24, 2025 @02:34PM (#44674373)
If someone released an open source software that immediately replaced the software you were working on and you lost your job, what then?

What if that happened across the entire industry overnight?

What if every single time you sat down to write a piece of software, someone immediately produced the Open Source version of it?

That's what people are hoping AI is going to do to us

Why would I support that

Re:The great AI delusion is falling apart by simonw on Thursday July 24, 2025 @03:15PM (#44674859)
> If someone released an open source software that immediately replaced the software you were working on and you lost your job, what then?

WordPress did that to the commercial CMS industry. Linux did that to the commercial Unix industry (RIP, Sun Microsystems).

> That's what people are hoping AI is going to do to us

I think those people are full of crap, and I mostly ignore them.

Re:The great AI delusion is falling apart by didibus on Thursday July 24, 2025 @01:51PM (#44673792)
I've always thought that I'd probably be paid even more and have the choice of many more jobs if open source didn't exist.

Could it be there'd be less jobs because it would have prevented some businesses to bootstrap and stay afloat? That's also possible.

And this is really the thing with AI, some think it'll grow the pie, therefore increasing the demand for developers. Some think it'll just shrink the size of the piece of pie that's for developers.

If we could predict the former, and be sure of it, I doubt anyone would be against AI.

Aside that though, as a developer, you are also concerned with, is this actually making my job better/faster/more-fun. Or am I just forced into an annoying process of using a tool that doesn't help and will just cause more problems I'll have to deal with.

Re:The great AI delusion is falling apart by simonw on Thursday July 24, 2025 @03:31PM (#44675000)
I am so confident that open source expanded the job market.

Prior to open source software development was awful. We spent all of our time re-inventing wheels - or, if we were lucky, licensing expensive badly shaped wheels from vendors and crossing our fingers that they would work (because we couldn't fix them if they didn't).

As a result, very few companies developed software - it's not worth hiring a software engineer if it takes them six months to deliver that initial login screen.

Open source started taking off about 25 years ago, and software productivity skyrocketed. The result was a multi-decade boom in jobs and salaries for us software engineers.

Re:The great AI delusion is falling apart by didibus on Thursday July 24, 2025 @06:17PM (#44676990)
It's plausible, I just want to point out it's hard to know if that's just coincidental, as opposed to causal.
Re:The great AI delusion is falling apart by thunky on Thursday July 24, 2025 @08:08PM (#44677881)
> We spent all of our time re-inventing wheels

Still do.

At least back then it probably felt more like we were doing something new because 8000 versions of the same thing wasn't right at your fingertips.

Re:The great AI delusion is falling apart by fragmede on Thursday July 24, 2025 @02:02PM (#44673942)
The thing about open source, is that it came from a programmer in the halls of academia, MIT. But imagine the idea came from a Harvard Business School MBA instead. "You should do a bunch of work for free, and then give it to me for free, and uh, you can figure out a business model later." Does that sound remotely sane?

I have no leg to stand on, I owe my career to proprietary software building on top of open source code, but lately I have been wondering what that alternate world would look like. As it stands, if you want to write software and sell it, you've got at most a couple of months before someone open sources a clone of your project. Sure, there's no guarantee it'll be anywhere near as good. Photoshop is still king. But one can't help but wonder what the proprietary IBM world would have been like.

Re:The great AI delusion is falling apart by simonw on Thursday July 24, 2025 @03:34PM (#44675036)
We lived in that world prior to ~2000. It sucked. Programmers spent all of their time reinventing the wheel - so much so that when I was at university around 2001-2004 one of the biggest topics was "software reusability". How could we get out of the trap of writing the same things over and over again?

Back then the answer was supposed to be OOP. With hindsight, the answer was open source - in particular robust library ecosystems like PyPI and NPM.

Re:The great AI delusion is falling apart by Yoric on Friday July 25, 2025 @04:02AM (#44680807)
Hey, I'm old enough to remember "components" being a research topic in academia, with researchers theorizing what we know today as services & APIs.

It looked more fun on paper, FWIW.

Re:The great AI delusion is falling apart by Yoric on Friday July 25, 2025 @04:00AM (#44680792)
Good point.

Open-source does replace entire companies, but typically with better products that other companies can build upon, and in the end, this expands the demand for software.

AI is expected to replace developers, and I believe that it will empower non-techies to develop highly-functional mockups, which will in turn increase the amount of (crappy) software on the market, which I believe will in turn increase the demand for people who can fix this software – if we multiply the number of apps by 1,000, even if only 1% of the apps need quality, this still means that we've increased the demand for software developers 10x.

So I actually think that, within a few years, AI will increase our job security as software developers. It's going to be a rocky few years, of course, but I think we'll be better off, at least until the rest of our dystopian timeline catches up.

Now, what I hate is the dishonesty of AI companies. I mean, as software developers, we strive on rigor: good documentation, compilers and JITs that speed up our code without changing its behavior, APIs and tools that work as claimed, etc. AI companies and their PR, on the other hand, peddle magic thinking and gaslighting. That's the exact opposite of rigor. And this keeps assaulting my sanity.

Re:The great AI delusion is falling apart by kbelder on Thursday July 24, 2025 @02:02PM (#44673953)
Should we be wary of any productivity gains?

Should be looking for ways to work slower? I can go back to just one monitor.

Re:The great AI delusion is falling apart by bluefirebrand on Thursday July 24, 2025 @02:26PM (#44674269)
If I personally do not benefit from productivity gains, then why should I be excited about them?

If productivity gains mean that either me or a coworker is laid off, and my boss makes more money, why should I care about the that?

Having a job is not a favor that people with money are doing for employees, it is an exchange of services for money

I provide a service. So do most people. Anything that gives the people with money a way to avoid paying for my service is bad for most of us

Re:The great AI delusion is falling apart by munksbeer on Friday July 25, 2025 @02:47PM (#44686777)
This is a slightly shallow way of looking at productivity gains. You benefit from productivity gains across every industry. Without consumers, there is no capital. So you, the consumer, benefit from increased productivity, even if it means that somewhere, someone, lost out. We just have to go back to the original luddite example. It is cliche for a reason. A quote:

"The tragedy is that although these groups of workers did suffer from the deployment of new technology, over time, the general population benefitted from cheaper clothes, lower prices and new jobs in other sectors of the economy."

Obviously, as a developer, I don't want to lose my career. But am I a hypocrite? Are you a hypocrite for buying anything that is now manufactured by automation, putting people out of jobs? I'd say yes.

Re:The great AI delusion is falling apart by ajaisjdh on Thursday July 24, 2025 @01:53PM (#44673822)
There’s literally billions of dollars on the “pro AI side”.

What you’re seeing is a thirst for objective reporting. The average person only has the ability to provide anecdotes - many of which are in stark contrast to the narrative pushed by the billionaires pumping AI.

I don’t think anyone serious thinks AI isn’t useful in some capacity - but it’s more like a bloom filter than a new branch of mathematics. Magically powerful in specific use cases, but not a paradigm shift.

Re:The great AI delusion is falling apart by didibus on Thursday July 24, 2025 @01:40PM (#44673659)
Personally, I think this is a disingenuous take. The thirst is for tangible data, the issue is that we've never been able to measure any form of productivity/quality in software development.

My team does two person PR reviews for example. We'd go a lot faster if we didn't or even just allowed a single reviewer. Similarly, we have no idea what the quality impact would be if we stopped, and what we gain by doing so. We are we not having a 3 reviewer rule for example, why not, two is an arbitrary number?

Unit tests... We'd surely go a lot faster if we didn't bother with them. Teams used to have some dedicated QA members and you'd rely entirely on manual testing. You can push a lot more code out. Was software in the 90s when unit tests and integ tests wasn't used buggier than today's software?

Now take AI, what is the impact of its use? It's not even obvious if it reduced the time it takes to launch a feature, my team isn't suddenly ahead of schedule on all our projects, even though we all use Agentic tools actively now. Ask any one of us and "I think it makes us faster" will be the answer. But ask us why we have a 2 person review rule and we'd similarly say: "I think it prevents bugs and improves the code quality".

The difference with AI now is that you pay for it, it's not free. Having unit tests or doing a 2 person review is just a process change. AI is something you pay for, so there's more desire to know for sure. And it also is something people would like to know if they can lower their headcount without impacting their competitive edge and ability to deliver fast and with good enough quality. Nobody wants to lower the headcount and find out the hard way.

Re:The great AI delusion is falling apart by thewebguyd on Thursday July 24, 2025 @01:56PM (#44673854)
> the issue is that we've never been able to measure any form of productivity/quality in software development.

Yep. It's been a "problem" for decades at this point. Business types constantly trying, and failing, to find some way to measure dev productivity, like they can with other types of office drone work.

We've been through Lines of Code, Function Points, various agile metrics, etc. None of these have given business types their holy grail of a perfectly objective measure of productivity. But no one wants to accept an answer of "You just can't effectively measure productivity in software development" because we now live in a data-driven business culture where every little thing must be measured and quantified.