I think the gap is quickly narrowing on the need to fully comprehend most 
  computer programs for most use cases, ...

I wish it could, but it appears not to.

As an example, ask it to code up something from a "less popular domain" (e.g., Clifford algebra).

You might notice that it appears to not have enough data to make good guesses. Some "AI" hype folks appear to move the goalpost then.

However, maybe the stock market and some companies benefit from all this sensationalist stuff.

I wish I had a HAL 9000 beside me, but I am sorry Dave, I cannot let you do that.

Obviously it could be completely wrong, but it certainly _seems_ to comprehend geometric algebra more than I do.

We are basically at the point where we could build the fembots from Ex Machina or David, the little boy from A.I., if we were just a little better at robotics. We'll see the teddy bear Supertoy from the latter movie under peoples' Christmas trees inside five years.

The ability to reason or understand seems to be not there in current LLMs.

See my other comment:

https://news.ycombinator.com/item?id=38819544

Simply put: "fake it 'till you make it" = probabilistic guesses given a large data set = seems to fool us people believing that an LLM "thinks" and "reasons".

There you go. Try it with 4. Still doesn't work? Let's revisit when 5 comes out. You will lose the argument eventually, it's only a question of when.

with deriving the geometric product (Clifford algebra) based on a set of definitions and axioms (e.g., the distributive property). Unfortunately, it failed, making numerous errors along the way.

Also maybe try asking it something that ordinary humans can be expected to accomplish, rather than complaining that nobody has invented Ramanujan-as-a-Service yet.

Also: Further, "AI" beating masters of chess also seems to be a product of "beat a human with probabilistic guesses given a large data set".

That was true for chess. Lee Se-Dol's defeat by AlphaGo was different. That wasn't supposed to happen. That was a combination of probabilistic Monte Carlo methods and... something else entirely.

That's really when the train of "AI can do X but not Y" left the tracks, IMO.

  There you go. Try it with 4. Still doesn't work? Let's revisit when 5 comes out. 

So, it "thinks" and "understands", it does not make (probabilistic) guesses?

Definition: guess

  to give an answer to a particular question when you do not have all the facts and so 
  cannot be certain if you are correct

To your other point:

  Also maybe try asking it something that ordinary humans can be expected to
  accomplish, rather than complaining that nobody has invented
  Ramanujan-as-a-Service yet.

  You will lose the argument eventually, it's only a question of when.

"To understand" and "to think" are two very different things. Understand means more or less to encode and compress effectively from a perspective of such a system - there's quite a bit of evidence that they do that.

As for "thinking" that is impossible for LLMs as thinking is an action - and LLMs aren't agents that can plan and take action.

Actually AlphaGO and AlphaZero were agents capable of thinking - just in a extremly simplistic world which is the game of Go, Shogi or Chess. But they had a world model (which was fully known as it was for simple games) and a way to plan the action they will take by evaluating what impact will they have upon the world and how beneficial it will be for them.

Just that extending that system/agent to the real world is very hard.

A flippant way to answer the question might be to ask, "What makes you so sure you're not arguing with an AI right now?"

A less-flippant answer is simply that we would need to agree on what understanding and thinking mean before we could debate whether AI is doing it now or can potentially do it in the future. I doubt we'll come to agreeable terms there, but I'd start by suggesting a simplistic hierarchy of cognition:

0) Memory: the ability to store information. Trivial.

1) Knowledge: the ability to retrieve that information associatively.

By that I'd imply a requirement that knowledge must be retrievable by indexing at a conceptual level. Knowledge isn't "I have four bananas and the third is ripe"; that's memory. Knowledge is "Bananas are edible fruits." It's more or less unquestionable that this is satisfied by embedding memorized items in a high-dimensional vector space where conceptual similarity corresponds to distance measured by some norm or another.

2) Understanding: the ability to cast new knowledge in terms of existing knowledge. This is where the increasingly-popular compression-as-intelligence metaphors start to come into play. Anytime you use an embedding scheme to compress data, you are exhibiting understanding. When Stable Diffusion encodes information about entire works of art with one or two bits of actual information, which it can later use to synthesize new works, that's "understanding" IMO.

3) Reasoning: I'm fine with the definitions of inductive reasoning (applying specific understanding to the solution of general problems) and deductive reasoning (deriving specific outputs from general understanding.) AI models perform induction whenever they recognize and encode patterns, and they perform deduction anytime they solve a specific problem using those patterns. At this point the burden of proof falls on anyone who claims these processes aren't happening.

So that leaves "thinking", which is a topic in philosophy rather than neuroscience or compsci, and forms of reasoning that go beyond the simplistic inductive and deductive models. Hypothesis formation is interesting, for instance. That's one area where someone using a GPT 3.x model might walk away with their negative preconceptions regarding AI being fully validated, as you did, while someone using GPT 4 might walk away with a furrowed brow and a dazed look. Maybe, once the previous bases are covered, Occam's Razor is all you need to perform abductive reasoning. If not, what else?

These are all interesting points of debate that go much deeper than "LOL, it can't even construct proofs using Clifford algebra." If you're satisfied with that, then again, there's not much room for the conversation to progress.

Writing fresh code is easy because you just express your ideas. What's hard is hitting all the contingencies appropriately; it can take years of 'burn in' to get right.

Reading code is easy - especially if you don't know the language well - because someone has already assembled all the correct syntax, etc. What's hard is that the code may be convoluted from years of monkey-patches to handle all those inconvenient contingencies, to the point of obscuring the core ideas. eg, making you search through ten nested indirections to figure out how something actually executes...

On the writing side, LLM's make it much easier to have the idiomatics of the language easily on hand. (eg, I never ever remember the exact way to use 'super' in python... but ML assistance is happy to fill in the parens as soon as I start them.)

What I’m getting from these comments is (as expected), most people are happy to take the easy assumption filled route rather than actually applying rigour.

We were a neuro lab, so coding was seen as something similar to a hammer: Bang away with it all you want, but at the end of the day, it's going back in a dark drawer where it belongs with the other tools.

English as a programming language makes this all work.

Some of the comments here actually freak me out.

It's kludge after hot patch after voodoo incantation. I once worked on a computer that under no circumstances could it be put on the web. It was running Windows 98 in ~2017 and was only used for a single program to control an instrument. I know for a fact that it's still online and running. It was slow as all get out, but no one had any kind of time, let alone gumption, let alone skill, to try to bring it up a millennium.

On old hardware/software, that's normal, and will continue to be normal, simply because the value is in the instrument, and money spent trying to get the software/drivers to work on more modern systems would be better spent on buying a new and better instrument. The only way this would change is if we force all hardware vendors to fully open-source (and ideally upstream) their code (look at the whole IoT ecosystem).

One still needs to find that balance between smart, functional / practical and maintainable. That’s the art of coding.