alecbz

I have a suspicion that the objective nature of most AI benchmarks is tapering over the kinds of reasoning mistakes LLMs continue to struggle with.

If you give LLMs a clearly defined spec with clear, objective acceptance criteria that the LLM can evaluate itself against, they tend to do well. But I think they struggle with more ambiguity and require a lot of handholding around judgment. That kind of disambiguating work is often baked in to the definition of a benchmark.

> The overwhelming majority of bad driving is caused by alcohol and other drugs, drowsiness, and inattentive driving.

> Sober, well-rested human beings who pay full attention to the road are surprisingly good drivers, to the point where I'm not actually sure that robotaxis can reliably beat them yet.

Including "inattentive" and "pay full attention to the road" feels a little tautological? Like, what's your definition of "inattentive"? More or less by definition, if there's an accident, someone was "inattentive" and didn't do the correct thing. Unless they were actively malicious and trying to hurt someone, which I agree would be a tiny minority of car accidents.

> that's primarily a human concern that's unrelated to the final output.

It's an LLM concern too. LLMs seem to do better with well-organized codebases, just like the humans they were trained on.

In the long-long-term, self-driving will get good enough and people will be relying on it so thoroughly that allowing people to drive on their own, with barely any practice, will probably be more risky than it's worth. In say, idk, 100 years (probably sooner), I doubt people will be able to drive their own cars in the way they do today.

Cab tipping culture is totally different and I would imagine varies significantly locally.

This is the study I was recalling: https://www.academia.edu/51293325/Restaurant_Tipping_and_Ser...

Though in looking for that I did find this other study that found racial bias: https://onlinelibrary.wiley.com/doi/abs/10.1111/soin.12056

FWIW it comes off as pretty snarky to provide evidence and then pre-emptively insult the reader by assuming they won’t respond to it, or assume that disagreement will be a result of rejecting evidence in general and not having some more specific objection (e.g., cab vs restaurant tipping culture being different)

Normalize price discrimination

Though actually I’d suspect a lot of that is just rising costs (rent)?

Knowing to flag down a waiter is probably a big part of the equation. In the US waiters will generally come on their own to take your order fairly quickly and so we’re less used to flagging them down. And in a lot of places in Europe waiters won’t ask to bring the check, it’s on you to ask for it.

At restaurants at least, most people I know just default tip 20% and only adjust up or down in pretty rare circumstances. Maybe it’s different in other parts of the US or for other generations, but I recall studies supporting that people generally have a set percent they always tip.

Software engineering has been getting more efficient for decades and over that time there’s been a massive increase in the number of software engineers.

The Mythical Man-Month describes teams of ten engineers operating before the Internet. Today (pre-LLMs) that’s probably 1-2 people.

> we won't even need programming languages

> just do “natural language” to a programming language

> Germany

The HRE for several hundred years could probably seen as a proto-Germany.

Interesting -- is there any term referring to the executive, legislative, and judicial functions collectively?

> At least the countries in the EU should have separation of powers, and the judicial (the courts) are independent from the government

In the US "government" generally refers to all three branches; is that not the case in Europe?

OOC what's your other income from?

Honestly, either feels like an improvement over now.

If LLMs actually get good enough to really automate the production of good software, it'll be disruptive for the industry and we'll all have to adjust a lot more than we already are, but I think it'd be on-net good for it to be cheaper and easier to produce good software. And, in the past, such changes have only increased the size of the tech industry.

Or, if we finally realize LLMs aren't going to get there, there'll be at least increased demand for actual software engineers to clean up all the LLM mess.

But right now is the worst, where the industry feels like it's lying to itself about what these tools are capable of.

Things like bad management, disillusionment with the company's goals, worries about being automated out of a job, etc. can lead people on the floor to care less and put in less effort.

There's centralized tooling for the PR descriptions, but I have some local flows where I try to provide more careful prompting and examples to get it to write better. It definitely helps but it's still not great and I'm often unsure if all the extra prompting is worth the effort.

This has unfortunately not been my experience at all. Often LLMs miss or get wrong subtle details when I don't do the pre-work to organize my thoughts well ahead of time (at which point it's unclear how much value they're providing).

The PR descriptions are pretty universally disliked. We have centralized tooling that manages the prompts for that, I’m sure they’ve tried tuning it but maybe there’s more they could do.

Though I have some local workflows where I try to teach Claude about my writing style preferences via skills and examples, and it’s still not great.

I mean, the “meta” right now is trying to automate as much of the software engineering flow as possible, including writing PR descriptions.

They’re shooting for LLMs being able to one-shot PRs or need minimal oversight. But yeah, in practice LLMs are not there IME.

LLM-generated (or at least LLM-assisted), but a human's still the one submitting the PR for review.

We have LLMs try to generate descriptions of PRs for us and they're pretty universally disliked. They're always overly-complex descriptions of the mechanical changes and have no sense of motivation.

Also, a huge reason to understand the code yourself is to make sure the LLM isn't wrong, but this doesn't work if an LLM is itself generating the understanding.

> understands people and knows what people need

> does not understand the concept of [incredibly common psychological problem]

I think if you just look at what people like e.g. Sam Altman are doing it's clear that they don't believe everything that they're saying regarding AI safety.

> nearly 100% of code would be written with AI in 2026

I feel like this is kind of a meaningless metric. Or at least, it's very difficult to measure. There's a spectrum of "let AI write the code" from "don't ever even look at the code produced" to "carefully review all the output and have AI iterate on it".

Also, it seems possible as time goes on people will _stop_ using AI to write code as much, or at least shift more to the right side of that spectrum, as we start to discover all kinds of problems caused by AI-authored code with little to no human oversight.

There's no need to tack on "because the wiki is open". That's still just part of a solution seeping into the statement of the problem.

I don't think this means "most concerns aren't valid", it's more "people aren't always good at vocalizing their underlying concerns, and instead treat a proposed solution as the concern".

Those sound like proposed solutions, not the underlying concerns. Motivating concerns here might be things like "our wiki will be full of inaccurate info", "people will unknowingly install spyware".

I don't know that you can establish objectively if someone is a good writer. He's an acclaimed, award-winning writer, sure.

Yeah I think the lesson is that specific suggestions for what to do aren't as helpful as just hearing how someone else experienced your work, and then drawing your own conclusions about how to fix that.

Bug reports should describe the problem but often shouldn't try to prescribe a solution.

How can you tell the difference?