I would paraphrase one of Clarke's laws and say that "Any sufficiently advanced text generator is indistinguishable from an intelligent entity."

Just because a computer program's output is remarkably good does not mean there is any emergent intelligence, any more than a technology we don't understand means there is magic.

The reverse can also be true, with John Keats' agreeing with Charles Lamb that Newton "had destroyed all the poetry of the rainbow, by reducing it to the prismatic colours": https://en.wikipedia.org/wiki/Lamia_(poem)

If we should ever fully understand how our own minds work, will we hold machines in higher esteem, or ourselves in lower?

Any biology or physics that suggests humans are just a pattern recognizer will be discarded as us being a conscious being is the only thing every human knows to be 100% true.

So, all of biology and physics then. If souls exist, they have no mass, and have a weird way of being repeatably disrupted in consistent ways by damage to certain parts of the brain or specific chemicals.

Just because consciousness is a mystery today, doesn't mean we get to stop and say it will be so forever more.

Heck, the problem still fundamentally exists regardless of if you're atheist, monotheist, polytheist, or pantheist.

--

“We’re not listening to you! You’re not even really alive!” said a priest.

Dorfl nodded. “This Is Fundamentally True,” he said.

“See? He admits it!”

“I Suggest You Take Me And Smash Me And Grind The Bits Into Fragments And Pound The Fragments Into Powder And Mill Them Again To The Finest Dust There Can Be, And I Believe You Will Not Find A Single Atom Of Life–”

“True! Let’s do it!”

“However, In Order To Test This Fully, One Of You Must Volunteer To Undergo The Same Process.”

There was silence.

“That’s not fair,” said a priest, after a while. “All anyone has to do is bake up your dust again and you’ll be alive…”

- Feet of Clay, Terry Pratchett

You missed the entire point. Physics and biology exist to help humans understand the material universe. Anything supposing that humans aren’t actually intelligent or conscious or whatever, or lack agency, is wrong since all of physics and biology are an offshoot of that agency meant to enrich it.

I'm not missing the point, I'm saying you're wrong. There's a difference.

Also:

> Anything supposing that humans aren’t actually intelligent or conscious or whatever

Doesn't really match what I was writing about: if it turns out that a thing which is "just a pattern recognizer" can in fact be "intelligent or conscious or whatever", it's up to us if we see intelligence or consciousness or whatever in the pattern recognisers that we build, or if we ourselves descend into solipsism and/or nihilism.

Or if we take the traditional path of sticking our fingers in our ears and go "la la la I'm not listening" by way of managing cognitive dissonance. This is a very popular response which should not be underestimated.

But the laws of physics are quite clear, that a whole bunch of linear equations (quantum field theory) gets us chemistry, which gets us biology, etc., and the only place in all this for the feeling of existence that we have is emergent properties. Those emergent properties may, or may not, be present in other systems, but we don't know because we're really bad at characterising how emergent properties… emerge.

This line of logic is more frightening to me than actual AI. LLMs are really useful in a lot of scenarios but it takes 5 minutes playing with one to see that it isn't intelligent.

But since you are the type of person who is seemingly using LLM "written" code in production, your ability to accurate assess anything is suspect at best.

"Any technology, sufficiently advanced, is indistinguishable from magic".

No, an LLM is not intelligent. I do not understand why people will go through mental gymnastics to conclude they are.

queue all the typical arguments supporting them being intelligent and demanding I give reasons for them not being

This is kind of a weird take... if you said your dog isn't intelligent because it can't do calculus and most people would look at you funny. You don't have to see your pet as intelligent, but don't expect everyone else to blindly follow your thinking.

s/queue/cue/ ;)

It’s not a human and so the entire argument comparing it to one is moot. It’s a program on a machine and doesn’t have rights, this anti-human way of thinking is seriously fucking scary.

The post you're responding to didn't call them human. Nor alive. Just "intelligent", and just as intelligence isn't required of life so I have no reason to think intelligence itself requires life.

These things are indeed "a program on a machine and doesn’t have rights", but what I find scary is that rights aren't part of the rules of the universe, they're merely laws, created and enforced (to the extent that they are at all) by humans.

As Max Tegmark said in his recent interview with Lex Fridman, a lot of the technology being developed now, and how it's being talked about and how it's used, is anti-life.