> I wonder if we've reached the limit of what a singular human mind can push.

You people are becoming zombies. Get away from your computer and interact with the world. Humans with profound intellect are producing the content that LLMs regurgitate for you.

Yes the LLM hype can be annoying.

Equally if not more annoying is the smug disdain of those who criticize it.

Even more annoying than that is people too caught up in "the world" and "serious matters" to read some sci-fi.

Don't get me wrong, I love reading and watching sci-fi! And actually, that's a great source of inspiration to keep my mind sharp; a superintelligence unleashed on our world today, with pathetically insecure networked listening devices in all of our pockets, a populace demonstrably manipulable through social media, etc; humanity has nothing but literal prayer to protect itself from actual superintelligence. Which isn't an actual defense, but hope feels nice.

Let's be clear: what we have today is LLMs. Not superintelligence. When people drool over the promise of superintelligence, to the exclusion of curiosity about the problems we intelligent humans can solve, they've bought into a thought-terminating utopian fantasy. That's a dead brain, that will not help humanity progress. Zombies.

Superintelligence, should it ever arise, is my enemy. The disdain that I have for its adherents is born of self-preservation.

edit to add clarity regarding my admonition to experience "the world": whoever receives all of their knowledge from computer sources is readily replaced by an automaton with access to the same. Those of us who are curious about the world around us are capable of making observations yet to be recorded in electronic form. Be curious, my friend, or succumb to zombiehood.

At the risk of repeating myself, yes the LLM hype is quite overdone and annoying. What you say is true people have bought into a fantasy which is wholly unrealistic.

Unfortunately your view is altogether tiresome and much harder to scroll past than the juvenile LLM excitement because it could actually do some real harm in the long term. No 'The Terminator" was not a documentary, nor was "The Matrix" nor even is your favorite episode of "Black Mirror".

Watts is far more eloquent than I could ever be on this subject matter. I highly recommend this article in The Atlantic by him and if you havent picked up any of his work I highly recommend pretty much anything hes ever written.

https://www.theatlantic.com/ideas/archive/2024/03/ai-conscio...

I was cured of blind techno-optimism years ago. The race for superintelligence is between ad-slinging megacorps and authoritarian governments. Whoever wins, we humans lose.

I Feel like were having two different conversations here. Who said anything about optimism? Are those the two options now optimism or doomerism? No space left for anything approaching realism?

I criticize blind techno-doomerism for its use of clanky old worn out cliches from the sci-fi of last century. I criticize LLM bros for being naive and blinded by advertising. Both groups completely misunderstood the relevant sci-fi from this century, if they actually bothered to read any.

Sometimes between the rabid hype-bros and the holier-than-thou doomers you can find actual conversation about the nature of human interaction with theoretical future machine intelligence, how LLMs relate to that and the technical aspects of creating such a thing. Its rare but it is out there.

You called me annoying because I don't read sci-fi. Now I stand accused of reading the wrong the wrong sci-fi? Whatever.

Hope for the best, prepare for the worst.

No one mentioned LLMs.

We know that there are things that computers can flat out do better than humans, because there are limits to human brains. If computers continue to grow in power, then the fact that someday there might be an AI (actual AI, not LLM) which can come up with novel solutions is totally plausible. And if we go past that point and computers continue to become more powerful, then it's likely that the AI will continue to grow and become more powerful as well. And if the AI eternally expands its thinking capabilities with time, then it will inevitably surpass humans one day.

Do we need to get to that point to solve physics? Probably not. We're not at the limits of human ingenuity yet. But mentioning AI doesn't mean "I think LLMs in the next 3 months will solve everything!" either. No need to be so defensive about someone wistfully pondering about the limits of humanity.

None of that is fact, or inevitable, or meaningful. What are "thinking capabilities", what are they made of? More RAM? A human can have more RAM, by installing it in the human's computer. More parallelism? Maybe, but that's like having more humans. More speed? It's not clear what you'd do with it, since research tends to involve interaction and the outside world has a pace of its own: sitting in a cell on amphetamines does not make you brilliant. More of some unknown component of the yet-to-be-discovered formula for intelligence? Maybe, if it even works that way, which is a massive assumption.

I'm sorry, I'm not buying "superintelligence", outside of sci-fi plots. Even there it's kind of irritating deus ex machina stuff.

I'd agree with what you're saying, but the jump to "not believing in superintelligence" doesn't make sense either. Are you implying that humans are the absolute peak of intelligence or that computers can't ever achieve something that is even just one iota better than human-level intelligence?

There's one thing machines have that we don't: time. A human-level AI would be better than a human because you can ask it to solve a problem for years on end without ever resting.

I'm implying that intelligence doesn't have quantifiable levels, so I disagree with "peak" and "better" as concepts. I know we can become better at solving problems in a domain, often with the assistance of tools. But I think the bogeyman of "superintelligence" is just a human with better tools. Perhaps including a synthetic brain. I expect its smarts to be cultural, made of ideas, mediated by technology, as usual, and domain-dependent, as usual, and not frighteningly "super" for the culture it exists in. I anticipate AGI as a development of us.

OK, so, maybe there are paradigm shifts to be reached that way, and maybe technology (familiarity with tools) will be crucial for these paradigm shifts which are the only way to understand certain new things intuitively. But, maybe this has been going on already, even before there were computers.

Humans do that too, ie research institutes.

I read their comment more as a nod to The Last Question than speaking about the recent "AI" in the news.

https://archive.org/details/Science_Fiction_Quarterly_New_Se...

https://en.wikipedia.org/wiki/The_Last_Question

Then they should've made an explicit reference. As is, I don't see any indication that they're even aware that The Last Question exists and it sounds like every other LLM fanatic.

The term AI has been around a heckuva lot longer than the current LLM hype. It's not unreasonable to make that assumption, to link it along the lines of what AI really means vs how the term has been used as of late.

Yes, it was more along the lines of The Last Question, wasn't even thinking about LLM.

The idea for me comes less from AI than the slowdown, I think, of really ground breaking science. We're still proving Einstein's ideas, we haven't really moved much beyond Einstein.

Thats the distinction I think between an LLM and a real AGI. One is a stochastic parrot, the other can come up with new ideas.

The distinction I focus on is that one exists, while the other remains a futuristic pipedream.

AI or not, if a machine finds an answer so complex that no human can understand it, how useful is it?

I don't see that as being an important objection, because from the point of view of any single human… that's already true for the expertise found only in other humans.

I don't know the chemistry necessary to turn crude oil into any plastic, I don't know how to use a Lagrangian, I tried and failed to learn group theory (I've only got the basics, and I'm not confident about them), and I've still only got a toy model of special relativity (which is supposed to be the easier one) — and yet, not only are those all still useful, the four colour theorem is useful even though it's a non-surveyable proof: https://en.wikipedia.org/wiki/Non-surveyable_proof

Depends on whether we can instruct the AI to make something useful out of it. Today we have complex systems like CPUs that most humans don't understand.

Or an answer so simple: 42.

Abstractions are a thing. We use devices all the time that we don't understand entirely the workings of.

It takes a singularity to comprehend a singularity?

Please touch grass sir!

> I wonder if we've reached the limit of what a singular human mind can push.

We passed that limit ages ago. The underlying technology used to do so is called "writing", which is what allows for things like libraries and letters / email and personal scratchpad notes.