Well it's a breadth & depth problem isn't it?

Humans are nonsensical, but in somewhat predictable error rates by domain, per individual. So you hire people with the skillsets, domain expertise, and error rates you need.

With an LLM, it's nonsensical in a completely random way prompt to prompt. It's sort of like talking into a telephone and sometimes Einstein is on the other end, and sometimes it's a drunken child. You have no idea when you pick up the phone which way its going to go.

We feed these things nearly the entirety of human knowledge, and the output still feels rather random.

LLMs have all that information and then still have a ~10% chance of messing up simple mathematical comparison that an average 12 year old would not.

Other times we delegate much more complex tasks to LLMs and they work great!

But given the nondeterminism it becomes hard to delegate tasks you can't check the work of, if it is important.

I haven't worked with LLMs enough to know this but I wonder: are they nonsensical in a truly random way or are they just nonsensical on a different axis in task space than normal humans, and we perhaps just haven't fully internalized what that axis is?

I'm not really sure, and you can pull lots of funny examples where various models have progress & regressions dealing with such mundane simple math.

As recently as August "11.10 or 11.9 which is bigger" came up with the wrong answer on ChatGPT and was followed with lots of wrong justification for the wrong answer. Even follow up math question "what is 11.10 - 11.9" gave me the answer "11.10 - 11.9 equals 0.2"

We can quibble about what model I was using, or what edge case I hit, or how quick they fixed it.. but this is 2 years into the very public LLM hype wave so at some point I expect better.

It gives me pause in asking more complex math questions I cannot immediately verify results, in which case, again why would I pay for a tool to ask questions I already know the answer to?

This error is not nonsensical though as normal elementary kids would make similar error and with good episodic memory the agent will fix itself.

He did say "sometimes Einstein is on the other end, and sometimes it's a drunken child. You have no idea when you pick up the phone which way its going to go.", so I think that's still a valid thing for him to complain about.

LLMs totally violate our expectations for computers, by being a bit forgetful and bad at maths.

Yes, to put a point on it -

How many dollars per month would someone be willing to spend for a chatbot that has a 3rd graders ability at math? Personally, $0 for me.

But what if it's a PHD Math degrees ability at math? Tons, in some applications it could be worth $100s or $1000s in an enterprise license setting.

But what if it's unpredictably, imperceptibly question to question, 95% PHD and 5% 3rd grader? Again, for me - $0. (not 95% of $1000s, but truly, $0)

Weirdly, I find myself agreeing with your vibes despite disagreeing on — oh, half? — the specifics.

I'm not sure what to make of that, but thought you might find it as curious as I do.

Thanks I think. My general vibe is - we are still confounding "wow this is neat" & "this is really useful" in this space. LLMs will lead to some real use cases & products.

I think we are seeing spaghetti against the wall / LLMs as a hammer right now. A lot of what we are seeing thrown out there is a misapplication of LLMs that will slowly fade away.. It is likely other techniques / models are required for some of the applications people are throwing LLMs at.

Feels reminiscent of "blockchain".

Yup, I agree with all of that.

Even though I see a lot of potential in the tech, it's still very obvious that people don't know how to use it to the best advantage and are hammering screws with it.