When someone says "AIs aren't really thinking" because AIs don't think like people do, what I hear is "Airplanes aren't really flying" because airplanes don't fly like birds do.

This really shows how imprecise a term 'thinking' is here. In this sense any predictive probabilistic blackbox model could be termed 'thinking'. Particularly when juxtaposed against something as concrete as flight that we have modelled extremely accurately.

Yes, to a degree. A very low degree at that.

AIs think like a rock flies.

If I shake some dice in a cup are they thinking about what number they’ll reveal when I throw them?

If I take a plane apart and throw all the parts off a cliff, will they achieve sustained flight?

If I throw some braincells into a cup alongside the dice, will they think about the outcome anymore than the dice alone?

that depends, if you explain the rules of the game you're playing and give the dice a goal to win the game, do they adjust the numbers they reveal according to the rules of the game?

If so, yes, they're thinking

The rules of the game are to reveal two independent numbers in the range [1,6].

As someone who plays a lot of Dungeons and Dragons, it sure feels like the dice are thinking sometimes.

That's a fallacy of denial of the antecedent. You are inferring from the fact that airplanes really fly that AIs really think, but it's not a logically valid inference.

Yeah at that point, just arguing semantics

Observing a common (potential) failure mode is not equivalent to asserting a logical inference. It is only a fallacy if you "P, therefore C" which GP is not (at least to my eye) doing.

"Observing a common (potential) failure mode"

That's not what we have here.

"It is only a fallacy if you "P, therefore C" which GP is not (at least to my eye) doing."

Some people are willfully blind.

Whenever someone paraphrases a folksy aphorism about airplanes and birds or fish and submarines I suppose I'm meant to rebut with folksy aphorisms like:

"A.I. and humans are as different as chalk and cheese."

As aphorisms are a good way to think about this topic?

ofc, and probably will never understand because of sheer complexity. It doesn't mean we can't replicate the output distribution through data. Probably when we do in efficient manners, the mechanisms (if they are efficient) will be learned too.

I agree we shouldn't undersell or underestimate the complexity involved, but when LLM's start contributing significant ideas to scientists and mathematicians, its time to recognize that whatever tricks are used in biology (humans, octopuses, ...) may still be of interest and of value, but they no longer seem like the unique magical missing ingredients which were so long sought after.

From this point on its all about efficiencies:

modeling efficiency: how do we best fit the elephant, with bezier curves, rational polynomials, ...?

memory bandwidth training efficiency: when building coincidence statistics, say bigrams, is it really necessary to update the weights for all concepts? a co-occurence of 2 concepts should just increase the predicted probability for the just observed bigram and then decrease a global coefficient used to scale the predicted probabilities. I.e. observing a baobab tree + an elephant in the same image/sentence/... should not change the relative probabilities of observing french fries + milkshake versus bicycle + windmill. This indicates different architectures should be possible with much lower training costs, by only updating weights of the concepts observed in the last bigram.

and so on with all other kinds of efficiencies.