I don't think this is a bad thing - if this can be accompanied by an increase in software quality, which is possible. Right now its very hit and miss and everyone has examples of LLMs producing buggy or ridiculous code. But once the tooling improves to:

1. align produced code better to existing patterns and architecture 2. fix the feedback loop - with TDD, other LLM agents reviewing code, feeding in compile errors, letting other LLM agents interact with the produced code, etc.

Then we will definitely start seeing more and more code produced by LLMs. Don't look at the state of the art not, look at the direction of travel.

That’s a huge “if”, and by your own admission not what’s happening now.

> other LLM agents reviewing code, feeding in compile errors, letting other LLM agents interact with the produced code, etc.

What a stupid future. Machines which make errors being “corrected” by machines which make errors in a death spiral. An unbelievable waste of figurative and literal energy.

> Then we will definitely start seeing more and more code produced by LLMs.

We’re already there. And there’s a lot of bad code being pumped out. Which will in turn be fed back to the LLMs.

> Don't look at the state of the art not, look at the direction of travel.

That’s what leads to the eternal “in five years” which eventually sinks everyone’s trust.

Humans are machines which make errors. Somehow, we got to the moon.

It is by the juice of Sapho that thoughts acquire speed, the lips become stained, the stains become a warning...

that "hallucinate" term is a marketing gimmick to make it seem to the gullible that this "AI" (i.e. LLMs) can actually think, which is flat out BS.

as many others have said here on hn, those who stand to benefit a lot from this are the ones promoting this bullcrap idea (that they (LLMs) are intelligent).

greater fool theory.

picks and shovels.

etc.

In detective or murder novels, the cliche is "look for the woman".

https://en.m.wikipedia.org/wiki/Cherchez_la_femme

in this case, "follow the money" is the translation, i.e. who really benefits (the investors and founders, the few), as opposed to who is grandly proclaimed to be the beneficiary (us, the many).

from a search for grand vs grandiose:

When it comes to bigness, there's grand and then there's grandiose. Both words can be used to describe something impressive in size, scope, or effect, but while grand may lend its noun a bit of dignity (i.e., “we had a grand time”), grandiose often implies a whiff of pretension.

https://www.merriam-webster.com/dictionary/grandiose

it's even more fundamental than that.

even if they had any model, they would not be able to think.

thinking requires consciousness. only humans and some animals have it. maybe plants too.

machines? no way, jose.

USA (s)election, I guess.

Even if we accept the premise (seeing humans as machines is literally dehumanising and a favourite argument of those who exploit them), not all machines are created equal. Would you use a bicycle to fill your taxes?

> Somehow, we got to the moon

Quite hand wavey. We didn’t get to the Moon by reading a bunch of text from the era then probabilistically joining word fragments, passing that around the same funnel a bunch of times, then blindly doing what came out, that’s for sure.

> The suggestion that errors just mindlessly compound and that there is no way around it

Is one that you made up, as that was not my argument.

We got to the moon using a large number of systems to a) avoid errors where possible and b) build in redundancies. Even an LLM knows this and knew what the statement meant:

https://chatgpt.com/share/6722e04f-0230-8002-8345-5d2eba2e7d...

Putting "corrected" in quotes and saying "death spiral" implies error compounding.

https://chatgpt.com/share/6722e19c-7f44-8002-8614-a560620b37...

These LLMs seem so smart.

Sure, I'm really poor painter, Midjourney is better than me. Are they better than a human trained for that task, on that task? That's the real question.

And I reckon the answer is currently no.

You’ve just proven my point. My issue with LLMs is precisely people turning off their brains and blindly taking them at face value, even arduously defending the answers in the face of contrary evidence.

If you’re basing your arguments on those answers then we don’t need to have this conversation. I have access to LLMs like everyone else, I don’t need to come to HN to speak with a robot.

Of course self-driving cars aren’t a nonsense idea. The execution and continued missed promises suck, but that doesn’t affect the idea. Claiming “humans are geniuses without equal” would be pretty dumb too, and is again something you’re making up. And something doesn’t have to be “all snake oil” to deserve specific criticism.

The world has nuance, learn to see it. It’s not all black and white and I’m not your enemy.

Actually understand LLMs in detail and you'll see it isn't some huge waste of time and energy to have LLMs correct outputs from LLMs.

Or, don't, and continue making silly, snarky comments about how stupid some sensible thing is, in a field you don't understand.

Yes, they do *seem* smart. My experience with a wide variety of LLM-based tools is that they are the industrialization of the Dunning-Kruger effect.

Now once you've recognized that, you're better equiped for task at hand - which is augmenting and ultimately automating away every task that humans-as-machines perform by building equivalent or better machine that performs said tasks at fraction of the cost!

People who want to exploit humans are the ones that oppose automation.

There's still long way to go, but now we've finally reached a point where some tasks that were very ellusive to automation are starting to show great promise of being automated, or atleast being greatly augmented.

If humans are machines, then so are fairies.

The conceit that humans are machines carries with it such powerful ideology: humans are for something, we are some kind of utility, not just things in themselves, like birds and rocks. How is it anything other than an affirmation of metaphysical/theological purpose to particularly humans? Why is it like that? This must be coming from a religious context, right?

I cannot at least see how you could believe this while sustaining a rational, scientific mind about nature, cosmology, etc. Which is fine! We can all believe things, just know you cant have your cake and eat it too. Namely, if anybody should believe in fairies around here, it should probably be you!

Because it's boring stuff, and most of us would prefer to be playing golf/tennis/hanging out with friends/painting/etc. If you look at the history of humanity, we've been automating the boring stuff since the start. We don't automate the stuff we like.

Recognizing that humans, just like birds are self-replicating biological machines is the most level-headed way of looking at it.

It is consistent with observations and there are no (apparent) contraditions.

The spritual beliefs are the ones with the fairies, binding of the soul, made of special substrate, beyond reason and understanding.

If you have desire to improve human condition (not everyone does) then the task at hand neturally arisies - eliminate forced labour, aging, disease, suffering, death, etc.

This all naturally leads to automation and transhumanism.

LLMs get their errors fed back into them and become more confident that their wrong code is right.

I'm not saying that's completely unsolvable, but that does seem to be how it works today.

Start adding different prompts, different models and you get all kinds of ways to catch errors. Just like humans.

No. LLMs can be told that there was an error and produce an alternative answer.

In fact LLMs can be told there was an error when there wasn't one and produce an alternative answer.

https://chatgpt.com/share/6722e41d-6b20-8002-8cbb-3012cd9179...

But a LLM is not "simply" doing anything. It's extremely complex and sophisticated. Once you go from tokens into high-dimensional embeddings... it seems these models (with enough training) figure out how all the concepts go together. I'd suggest reading the word2vec paper first, then think about how attention works. You'll come to the conclusion these things are likely to be able to beat humans at almost everything.

In my experience, if you confuse an LLM by deviating from the the "expected", then all the shims of logic seem to disappear, and it goes into hallucination mode.

ChatGPT here went with a lexigraphical order in Python for some reason, and then proceeded to make false statements from false observations, while also defying its own internal logic.

    "six" > "ten" is true because "six" comes after "ten" alphabetically.

    "ten" > "seven" is false because "ten" comes before "seven" alphabetically.

From what I understand of LLMs (which - I admit - is not very much), logical reasoning isn't a property of LLMs, unlike information retrieval. I'm sure this problem can be solved at some point, but a good solution would need development of many more kinds of inference and logic engines than there are today.

Really? That must be a very recent development, because so far this has been a reason for not using them at scale. And noone is.

Do you have a source?

There was a recent meme about asking LLMs to draw a wineglass full to the brim with wine.

Most really struggle with that instruction. No matter how much you ask them to correct themselves they can’t.

I’m sure they’ll get better with more input but what it reveals is that right now they definitely do not understand their own output.

I’ve seen no evidence that they are better with code than they are with images.

For instance, if the time to complete only scales with length of the token and not the complexity of its contents then it probably safe to assume it’s not being comprehended.

Are you sure you wanted to say that? Or is the other way around?

Indeed, and one of the most interesting errors some human machines are making is hallucinating false analogies.

It's an insanely conservative tool

Lastly, it’s also a huge waste of energy to feed the same information over and over again for each query.

- context over training is like someone referencing docs vs vaguely recalling from decayed memory

- context caching

The idea that our current languages might be as far as we get is absolutely demoralising. I don't want a tool to help me write pointless boilerplate in a bad language, I want a better language.

The companies valuing the expensive talent currently working on Google will be the winner.

Google and others are betting big right now, but I feel the winner might be those who watches how it unfolds first.

As a dev, I like it. It speeds up writing easy but tedious code. It's just a bit smarter version of the refactoring tools already common in IDEs...

Of course LLMs are a fantastic tool to improve productivity, but current LLM's cannot produce anything novel. They can only reproduce what they have seen.

And if we get 9 women we can produce a baby in a single month.

There's no guarantee such progression will continue. Indeed, there's much more evidence it is coming to a a halt.

More and more of the content generated since is LLM generated and useless as training data.

The models get worse, not better by being fed their own output, and right now they are out of training data.

This is why Reddit just went profitable, AI companies buy their text to train their models because it is at least somewhat human written.

Of course, even reddit is crawling with LLM generated text, so yes. It is coming to a halt.

It might be possible but will shareholders/investors foot the bill for the 80% that they still have to pay.

Why no youtube videos thought? Well, most dev you tubers are actual devs that cultivate an image of "I'm faster than LLM, I never re-read library references, I memorise them on first read" and do on. If they then show you a video how they forgot the syntax for this or that maven plugin config and how LLM fills it in 10s instead of a 5min Google search that makes them look less capable on their own. Why would they do that?

It is now very easy to sprinkle in regexes to validate user input , like email addresses, on every controller instead of using a central lib/utility for that.

In the hands of a skilled engineer it is a good tool. But for the rest it mainly serves to output more garbage at a higher rate.

Some people are touting this as a major feature. "I don't have to pull in some dependency for a minor function - I can just have AI write that simple function for me." I, personally, don't see this as a net positive.

The way it is now just leads to bloat and cruft.

Maybe I'm just old, but to me, LLMs feel like magic. A decade ago, anyone predicting their future capabilities would have been laughed at.

And so were a lot of markov chain based chatbots. Also Doretta, the microsoft AI/search engine chatbot.

Were they as good? No. Is this an iteration of those? Absolutely.

The discussion here seems to bare this out: CEO claims AI is magical, here the truth becomes that it’s just an auto-complete engine.

At Google, today, for sure.

I do believe we still are not across the road on this one.

> if this can be accompanied by an increase in software quality, which is possible. Right now its very hit and miss

So, is it really a smart move of Google to enforce this today, before quality have increased? Or did this set off their path to losing market shares because their software quality will deteriorate further over the next couple years?

From the outside it just seems Google and others have no choice, they must walk this path or lose market valuation.

That's what people are doing. The direction of travel over the most recent few (6-12) months is mostly flat.

The direction of travel when first introduced was a very steep line going from bottom-left to top-right.

We are not there anymore.

I'm excited about the possibilities and I still recoil at the refined marketer prose.

For teams you can measure meaningful outcomes and improve team metrics.

You shouldn’t really compare teams but it also is possible if you know what teams are doing.

If you are some disconnected manager that thinks he can make decisions or improvements reducing things to single numbers - yeah that’s not possible.

How? Which metrics?

Then I would have to go on outlining 6-12 months of trying stuff out.

Because if I just give "an example" I will get dozens of "smart ass" replies how this specific one did not work for them and I am stupid. Thanks but don't have time for that or for writing an essay that no one will read anyway and call me stupid or demand even more explanation. :)

This seems to be useful to understand and internalize that there are no simple answers like "use story points!".

There is also loads of people who don't understand that, so I stand by that is useful and important to repeat on every possible occasion.

Measuring it is not the hard part.

The hard part is doing anything about it. If you can't attribute specific outputs to specific inputs, you don't know how to change inputs to maximize outputs. That's what managers need to do, but of course they're often just guessing.

But I do code myself, I write requirements so I do know which ones are trivial and which ones are not. I also see when there are complex migrations.

If you work in a group of people you will also get feedback - doesn't have to be snitching but still you get the feel who is a slacker in the group.

It is hard to quantify the output if you want to be removed from the group "give me a number" manager. If you actually do the work of a manager so you get the feel of the group like who is "Hermione Granger" nagging that others are slacking and disregard their opinion, you see who is the "silent doer" or you see who is "we should do it properly" bullshitter you can make a lot of meaningful adjustments.

"You can [accurately and meaningfully measure software engineering productivity] - but not on the level of a single developer and you cannot use those measures to manage productivity of a specific dev."

At the level of a company like Google, it's easy: both inputs and outputs are measured in terms of money.

I am not Amazon person - but from my experience 2 pizza teams was what worked and I never implemented it myself just what I observed in wild.

Measuring Google in terms of money is also flawed, there is loads of BS hidden there and lots of people paying big companies more just because they are big companies.

So that's how animal husbandry came about!

Even that would be hard since hunting is complex. If you are the one chasing the pray into the arms of someone else, you surely want it to be considered a team effort.

You need like 'blueberries picked'.

I think it's harder to measure things like developer productivity. The closest thing we have is making an estimate and seeing how far off you are, but that doesn't account for hedging estimates or requirements suddenly changing. Changing requirements doesn't matter for DORA as it's just another sample to test for deployment.

Unfortunately there's a lot of lag

A great generalisation and understatement! Often looking like you are becoming more efficient is more important than actually being more efficient, e.g you need to impress investors. So you cut back on maintenance and other cost centres and the new management can blame you in 6 years time for it when you are far enough away from it to not hurt you.

edit: typo

That is, I, personally, am not measured on how much AI generated code I create, and while the number is non-zero, I can't tell you what it is because I don't care and don't have any incentive to care. And I'm someone who is personally fairly bearish on the value of LLM-based codegen/autocomplete.

None of this works to evaluate individuals or even teams. But it can be effective at evaluating tools.

To use your example, a user with an LLM might say "LLM please fix this" as a first line of action, drastically improving this metric, even if it ruins your overall productivity.

The superstar developer’s secret… he would send blank reports to clients (who would only realize it days later, and someone else would end up redoing the report), and he would score many more points without doing anything. I’ve seen this happen a lot in many different companies. As a friend of mine used to say, “it’s very rare, but it happens all the time.”

I have no doubt that AI can help developers, but I don’t trust the metrics of the CEO or people who work on AI, because they are too involved in the subject.

1) They can work to improve the system

2) They can distort the system

3) Or they can distort the data

https://commoncog.com/goodharts-law-not-useful/

ah, to be young again...

Now, I was really bad at capitalizing on it, so nothing much came of it, but still, there are some positive things that higher-ups do notice.

I’m far from impressed with the output of GPT/Claude, all they’ve done is weight against stack overflow - which is still low quality code relative to Google.

What is probability Google makes this a real product, or is it too likely to autocomplete trade secrets?

Will AI be able to detect bugs and back doors that require multiple pieces of code working together rather than being in a single piece of code? Humans have a hard time with this.

- Hypothetical Example: Authentication bugs in sshd that requires a flaw in systemd which then requires a flaw in udev or nss or PAM or some underlying library ... but looking at each individual library or daemon there are no bugs that a professional penetration testing organization such as the NCC group or Google's Project Zero would find. In other words, will AI soon be able to find more complex bugs in a year than Tavis has found in his career and will they start to compete with one another and start finding all the state sponsored complex bugs and then ultimately be able to create a map that suggests a common set of developers that may need to be notified? Will there be a table that logs where AI found things that professional human penetration testers could not?

Adversaries are already detecting issues tho, using proven means such as code review and fuzzing.

Google project zero consists of a team of rock star hackers. I don't see LLM even replacing junior devs right now.

Considering how terrible and frequently broken the code that the public facing Gemini produces, I'll have to be honest that that kind of scares me.

Gemini frequently fails at some fairly basic stuff, even in popular languages where it would have had a lot of source material to work from; where other public models (even free ones) sail through.

To give a fun, fairly recent example, here's a prime factorisation algorithm it produce for python:

  # Find the prime factorization of n
  prime_factors = []
  while n > 1:
    p = 2
    while n % p == 0:
      prime_factors.append(p)
      n //= p
    p += 1
  prime_factors.append(n)

You were probably being rhetorical, but there are two problems:

- `p = 2` should be outside the loop

- `prime_factors.append(n)` appends `1` onto the end of the list for no reason

With those two changes I'm pretty sure it's correct.

    def factorize(n):
      ...
      return prime_factors

Assuming discussions don't happen in Slack, or Discord, or...

Did you know? How WEIRD.

How about you not harass other commenters with such arrogantly ignorant sarcastic questions?? Or is that part of corporate "for-profit" culture too????

So is marketing? So finance? So is petroleum engineering?

I bet I'd make more errors on my first try at it.

If that doesn't satisfy, here's a similar one at leetcode.com: https://leetcode.com/problems/distinct-prime-factors-of-prod...

I would not expect a programmer of any seniority to churn stuff like that and have it working without testing.

I've been able to write one, not from memory but from first principles, any time in the last 40 years.

I'm happy you know math, but my point before this thread got derailed was that we're holding (coding) AI to a higher standard than actual humans, namely to expect to write bug-free code.

This seems like a very layman attitude and I would be surprised to find many devs adhering to this idea. Comments in this thread alone suggests that many devs on HN do not agree.

Testing for small prime factors is easy - brute force is your friend. Testing for large prime factors requires more effort. So the first trick is to figure out the bounds to the problem. Is it int32? Then brute-force it. Is it int64, where you might have a value like the Mersenne prime 2^61-1? Perhaps it's time to pull out a math reference. Is it longer, like an unbounded Python int? Definitely switch to something like the GNU Multiple Precision Arithmetic Library.

In this case, the maximum value is 1,000, which means we can enumerate all distinct prime values in that range, and test for its presence in each input value, one one-by-one:

    # list from https://www.math.uchicago.edu/~luis/allprimes.html
    _primes = [
        2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59,
        61, 67, 71, 73, 79, 83, 89, 97, 101, 103, 107, 109, 113, 127, 131,
        137, 139, 149, 151, 157, 163, 167, 173, 179, 181, 191, 193, 197,
        199, 211, 223, 227, 229, 233, 239, 241, 251, 257, 263, 269, 271,
        277, 281, 283, 293, 307, 311, 313, 317, 331, 337, 347, 349, 353,
        359, 367, 373, 379, 383, 389, 397, 401, 409, 419, 421, 431, 433,
        439, 443, 449, 457, 461, 463, 467, 479, 487, 491, 499, 503, 509,
        521, 523, 541, 547, 557, 563, 569, 571, 577, 587, 593, 599, 601,
        607, 613, 617, 619, 631, 641, 643, 647, 653, 659, 661, 673, 677,
        683, 691, 701, 709, 719, 727, 733, 739, 743, 751, 757, 761, 769,
        773, 787, 797, 809, 811, 821, 823, 827, 829, 839, 853, 857, 859,
        863, 877, 881, 883, 887, 907, 911, 919, 929, 937, 941, 947, 953,
        967, 971, 977, 983, 991, 997]

    def distinctPrimeFactors(nums: list[int]) -> int:
        if __debug__:
            # The problem definition gives these constraints
            assert 1 <= len(nums) <= 10_000, "size out of range"
            assert all(2 <= num <= 1000 for num in nums), "num out of range"

        num_distinct = 0
        for p in _primes:
            for num in nums:
                if num % p == 0:
                    num_distinct += 1
                    break
        return num_distinct

    import unittest

    class TestExamples(unittest.TestCase):
        def test_example_1(self):
            self.assertEqual(distinctPrimeFactors([2,4,3,7,10,6]), 4)

        def test_example_2(self):
            self.assertEqual(distinctPrimeFactors([2,4,8,16]), 1)

        def test_2_is_valid(self):
            self.assertEqual(distinctPrimeFactors([2]), 1)

        def test_1000_is_valid(self):
            self.assertEqual(distinctPrimeFactors([1_000]), 2) # (2*5)**3

        def test_10_000_values_is_valid(self):
            values = _primes[:20] * (10_000 // 20)
            assert len(values) == 10_000
            self.assertEqual(distinctPrimeFactors(values), 20)

    @unittest.skipUnless(__debug__, "can only test in debug mode")
    class TestConstraints(unittest.TestCase):
        def test_too_few(self):
            with self.assertRaisesRegex(AssertionError, "size out of range"):
                distinctPrimeFactors([])
        def test_too_many(self):
            with self.assertRaisesRegex(AssertionError, "size out of range"):
                distinctPrimeFactors([2]*10_001)
        def test_num_too_small(self):
            with self.assertRaisesRegex(AssertionError, "num out of range"):
                distinctPrimeFactors([1])
        def test_num_too_large(self):
            with self.assertRaisesRegex(AssertionError, "num out of range"):
                distinctPrimeFactors([1_001])

    if __name__ == "__main__":
        unittest.main()

If I had no internet access, I would compute that table thusly:

  _primes = [2]
  for value in range(3, 1000):
    if all(value % p > 0 for p in _primes):
        _primes.append(value)

What kind of senior programmers do you work with who can't handle something like this?

EDIT: For fun I wrote an implementation based on sympy's integer factorization:

    from sympy.ntheory import factorint
    def distinctPrimeFactors(nums: list[int]) -> int:
        distinct_factors = set()
        for num in nums:
            distinct_factors.update(factorint(num))
        return len(distinct_factors)

        def test_Mersenne(self):
            self.assertEqual(distinctPrimeFactors(
                [2**44497-1, 2,4,3,7,10,6]), 5)

Perhaps even, there could be fewer Go programmers here and some just took a stab at it even though they don’t know the language. So it could just select for which forum has the most Go programmers. Hardly rigourous.

So I’d take that with a pinch of salt personally

But I don't understand this attempt to tell companies/persons that are successfully using AI that no they really aren't.

In my opinion, if they feel they're using AI successfully, the goal should be to learn from that.

I don't understand this need to tell individuals who say they are successfully using AI that, "no you aren't."

It feels like a form of denial.

Like someone saying, "I refuse to accept that this could work for you, no matter what you say."

The whole paradigm should change. If you are indeed responsible for developer tools, I would hope that you’re activity leveraging Claude 3.5 Sonnet and o1-preview.

It's always felt like having AI in the cloud for better autocomplete is a lot for a small gain.