It’s possible that this could happen but you need to propose a mechanism and metric for this argument to be taken seriously (and to avoid fooling yourself with moving goalposts). Under what grounds do you assert that the trend line will stop where you claim it will stop?
Yes, if super-human AGI simply never happens then the alignment problem is mostly solved. Seems like wishful thinking to me.
The standard electrical wall sockets that you use have not really changed since WW2. For load bearing elements in buildings, we don't have anything substantially better today than 100 years ago. There is a huge list of technological items where we've polished out almost every last wrinkle and a 1% gain once a decade is hailed as miraculous.
Yes...
> Argue the facts.
What?
> What are the limits that prevent progress on the dimension of replicating human intelligence?
I don't work in that field, but as a layman I'd wager the lack of clear technical understanding of what animal intelligence actually is, let alone how it works is the biggest limitation.
Half of the language in your comment was fact free and incendiary. Delusion, prattle, deep needs of some role. That’s all firmly in the realm of ad hominem, so I was asking if you had anything substantial.
> I'd wager the lack of clear technical understanding of what animal intelligence actually is
I made the same points long time ago so can’t be too critical of that. I’ve changed my mind and here’s why. That’s not any kind of universal limit. It’s a state, and we can change states. We currently don’t understand intelligence but there’s no barrier I’m aware of that prevents progress. In addition, we’ve discovered types of intelligence (LLM’s, AlphaGo Zero etc) that don’t depend on our ability to understand ourselves. So our inability to understand intelligence isn’t a limit that prevents progress. New algorithms and architectures will be tested, in an ongoing fashion, because the perceived advantages and benefits are so great. It’s not like the universe had a map for intelligence before we arrived, it emerged from randomness.
I’m less sure that it’s a good idea, but that’s a different discussion. Put me in the camp of “this is going to be absolutely massive and I wish people took it more seriously.”
No. I believe it is factually accurate that many tech workers believe that technology progress marches steadily onwards and upwards. This is easily shown by the historical record to be patently false. Moreover, it's easy to imagine myriad ways we could regress technologically--nuclear war, asteroid impact, space weather, etc. So a belief in the inexorable progress of technology is therefore delusional. I believe it is a fact that tech companies' managers encourage these delusions by spinning up a bunch of pseudo religious sentiment using language like "innovation" to describe what is mostly actually really mundane, meaningless, and often actively harmful work. People hear that stuff, it goes to their heads, and they think they're "making the world a better place". The inexorable march of technology progress fits into such a world view.
> We currently don’t understand intelligence but there’s no barrier I’m aware of that prevents progress. In addition, we’ve discovered types of intelligence (LLM’s, AlphaGo Zero etc) that don’t depend on our ability to understand ourselves.
How can you claim both that we don't know what intelligence is, and that LLMs, AlphaGo Zero, and etc are "intelligent"?
There are many reasons for all those things not to change. Limits abound. We discovered that getting taller or faster isn’t “better”, all we needed is smarter. Intelligence is different. It applies to everything else. You can lose a limb or eyesight and still be incredibly capable. The intelligence is what makes us able to handle all the other limits and change the world even though MS Word hasn’t changed much.
We are now applying a lot of our intelligence to inventing another one. The architecture won’t stay the same, the limits won’t endure. People keep trying and it’s infinitely harder to imagine reasons why progress will stop. Just choose any limit and defend it.
What about post-tensioned concrete slabs? Cross-laminated timber? Structural glazing? Intumescent paint?
Technology does keep improving, actually.
I can also create a web scale app in a weekend using AWS. It is just insane what we can do now vs. 1999. I remember in early 2000s Microsoft boasting how it could host a site for the olympics using active server pages. This was PR worthy. That would be a side project for most of us now using our pocket money.
Same for video calls. Screen sharing can be useful at times, but it would be easy distribute materials to all participants and collaborate on a conference call. You'll have better audio latency that way. So it's not super obvious to me that the latest Wunderwaffe are actually significantly better, but it is clear they use a hell of a lot more compute cycles and bandwidth.
AI can magically decide where to put small pieces of code. Its not a leap to imagine that it will later be good at knowing where to put large pieces of code.
I don't think it'll get there any time soon, but the boundary is less crisp than your metaphor makes it.
Magically, but not particularly correctly.
Right.
It sounds to me like you agree and are repeating the comment but are framing as disagreeable.
I'm sure I'm missing something.
I tend to agree with them. What people seem to miss about LLM coding systems, IMO:
a) deciding on the capabilities of an LLM to code after a brief browser session with 4o/claude is comparable to waking up a coder in the middle of the night, and having them recite the perfect code right then and there. So a lot of people interact with it that way, decide it's meh, and write it off.
b) most people haven't tinkered with with systems that incorporate more of the tools human developers use day to day. They'd be surprised of what even small, local models can do.
c) LLMs seem perfectly capable to always add another layer of abstraction on top of whatever "thing" they get good at. Good at summaries? Cool, now abstract that for memory. Good at q/a? Cool, now abstract that over document parsing for search. Good at coding? Cool, now abstract that over software architecture.
d) Most people haven't seen any RL-based coding systems yet. That's fun.
----
Now, of course the article is perfectly reasonable, and we shouldn't take what any CEO says at face value. But I think the pessimism, especially in coding, is also misplaced, and will ultimately be proven wrong.
This is a good question, and I worry you won't get a response. Here is a pattern I've observed very frequently in the LLM space, with much more frequency than random chance would suggest:
Bob: "Oh, of course it didn't work for you, you just need to use an ANA (amazing new acronym) model"
Alice: "Oh, that's great, where can I see how ANA works? How do I use it?"
** Bob has left the chat **However, I worry about the premise underlying your reply: a sense that this is somehow incompatible with the viewpoint being discussed.
i.e. it's perfectly cromulent to both think LLMs are and will continue to be awesome at coding, even believe they get much better. And also that you could give me ASI today, and there'd be an incredible long tail of work and processes to reformulate to pull off replacing most labor. It's like having infinite PhDs available by text message. Unintuitively, not that much help. I can't believe I'm writing that lol. But here we are.
Steve Sinfosky had a good couple long posts re: this on X that discuss this far better than I can.
If you had a printer that could print semi-random mechanical parts, using it to make a car would be obviously dumb, right? Maybe you would use it to make, like, a roller blade wheel, or some other simple component that can be easily checked.
While the attention-based mechanisms of the current generation of LLMs still have a long way to go (and may not be the correct architecture) to achieve requisite levels of spatial reasoning (and of "practical" experience with how different shapes are used in reality) to actually, say, design a motor vehicle from first principles... that future is far more tangible than ever, with more access to synthetic data and optimized compute than ever before.
What's unclear is whether OpenAI will be able to recruit and retain the talent necessary to be the ones to get there; even if it is able to raise an order of magnitude more than competitors, that's no guarantee of success. My guess would be that some of the decisions that have led to the loss of much senior talent will slow their progress in the long run. Time will tell!
The invention would never see the light of day. If someone were to invent Star Trek replicators, they'd be buried along with their invention. Best Case it would be quickly captured by the ownership class and only be allowed to be used by officially blessed manufacturing companies, and not any individuals. They will have learned their lesson from AI and what it does to scarcity. Western [correction: all of] society is hopelessly locked into and dependent on manufacturing scarcity, and the idea that people have to pay for things. The wealthy and powerful will never allow free abundance of physical goods in the hands of the little people.
So to solve this problem you need billions to burn on gambles. I guess that's how we ended up with VC's.
How do you reconcile that with the fact that Western society has invented, improved, and supplied many of the things we lament that other countries don't have (and those countries also lament it - it's not just our own Stockholm Syndrome).
Are there specific historical examples of this that come to mind?
But I don't know of anything nearly as extreme as destroying an entire invention. Those tend to stick around.
There are a number of counterexamples though. Henry Ford, etc.
It’s rather hard to imagine even something like that (and it’s pretty limited in scope compared to the grand conspiracy above) working today, though; the EC would definitely stomp on it, and even the sleepy FTC would probably bestir itself for something so blatant.
In reality, the biggest problem was they had no incentive to invest in new lighting technology research, although they had the money to do so. It takes a lot of effort to develop a new technology, and significantly more to make it practical and affordable.
I think the story of the development of the blue LED which led to modern LED lighting is more illustrative of the real obstacles of technological development.
Companies/managers don't want to invest in R&D bc it’s too uncertain and they typically are more interested in the short term.
And it’s hard for someone without deep technical knowledge to identify a realistic worthwhile technical idea from a bad one. So they focus on what they can understand and what they can quantify ().
And even technical people can fail to properly evaluate ideas that are even slightly outside their area of expertise (or even sometimes the ones that are within it )
Then there would be a violent revolution which wrestles it out of their hands. The benefits of such a technology would be immediately obvious to the layman and he would not allow it to be hoarded by a select few.
Learning how to get what you want is a fundamental skill you start learning from infancy.
I think the insight is that some people truly believe that LLMs would be exactly as groundbreaking as a magical 3D printer that prints out any part for free.
And they're pumping AI madly because of this belief.
We have a very limited ability to define human intelligence, so it is almost impossible to know how near or far we are from simulating it. Everyone here knows how much a challenge it is to match average human cognitive abilities in some areas, and human brains run at 20 watts. There are people in power that may take technologists and technology executives at their word and move very large amounts of capital on promises that cannot be fulfilled. There was already an AI Winter 50 years ago, and there are extremely unethical figures in technology right now who can ruin the reputation of our field for a generation.
On the other hand, we have very large numbers of people around the world on the wrong end of a large and increasing wealth gap. Many of those people are just hanging on doing jobs that are actually threatened by AI. They know this, they fear this, and of course they will fight for their and their families lifestyles. This is a setup for large scale violence and instability. If there isn't a policy plan right now, AI will be suffering populist blowback.
Aside from those things, it looks like Sam has lost it. The recent stories about the TSMC meeting, https://news.ycombinator.com/item?id=41668824, was a huge problem. Asking for $7T shows a staggering lack of grounding in reality and how people, businesses, and supply chains work. I wasn't in the room and I don't know if he really sounded like a "podcasting bro", but to make an ask of companies like that with their own capital is insulting to them. There are potential dangers of applying this technology; there are dangers of overpromising the benefits; and neither of them are well served when relatively important people in related industries thing there is a credibility problem in AI.
The problem is when the hype machine causes the echoes to replace the original intelligence that spawned the echoes, and eventually those echoes fade into background noise and we have to rebuild the original human intelligence again.
As I said, it is overhyped in some areas and underhyped in others.
But what's interesting when I speak to laymen is that the hype in the general public seems specifically centered on the composite solution that is ChatGPT. That's what they consider 'AI'. That specific conversational format in a web browser, as a complete product. That is the manifestation of AI they believe everyone thinks could become dangerous.
They don't consider the LLM API's as components of a series of new products, because they don't understand the architecture and business models of these things. They just think of ChatGPT and UI prompts (or it's competitor's versions of the same).
*(which is always a risky way of looking at it, because who the hell am I? Neither somebody in the AI field, nor completely naive toward programming, so I might be in some weird knows-enough-to-be-dangerous-not-enough-to-be-useful valley of misunderstanding. I think this describes a lot of us here, fwiw)
An AGI however could. Once it reaches IQs of more than say 500 it would become very hard to control it.
However I think it’s more likely that it will LARP as “I’m an emotionally supportive beautiful AI lady, please download me to your phone, don’t take out the battery or I die!”
That was part of the plot of person of interest. A really interesting show, it started as a basic "monster of the week" show but near the end it became a much more interesting plot.
Although most of the human characters were extremely one-dimensional. Especially Jim Caviezel's who was just a grumpy super soldier in every episode. It was kinda funny because they called him "the man in the suit" in the series and there was indeed little else to identify his character. The others were hardly better :(
But the AI storyline I found very interesting.
[0]: No Physical Substrate, No Problem https://slatestarcodex.com/2015/04/07/no-physical-substrate-...
[1]: It Looks Like You’re Trying To Take Over The World https://gwern.net/fiction/clippy
Why is it in your worldview a CEO “has to lie”?
Are you incapable of imagining one where a CEO is honest?
> The real story of LLMs is revealed when you posit a magical technology that can print any car part for free.
I’ll allow it if you stipulate that randomly and without reason when I ask for an alternator it prints me toy dinosaur.
> It is easy to imagine that the inventor of such a technology
As if the unethical sociopath TFA is about is any kind of, let alone the, inventor of genai.
> Being able to generate code/text/images is of no use to someone who doesn't know what to do with it.
Again, conveniently omitting the technology’s ever present failure modes.
I'm not talking about Altman in particular, I'm just annoyed with this constant spam on HN about how we all need to turn a blind eye to snake oil salesman because "that's just how it's supposed to be for a startup."
For a forum that complains about how money ruins everything, from the Unity scandal to OSS projects being sponsored and "tainted" by "evil companies," it's shocking to see how often golden boy executives are excused. I wish people had this energy for the much smaller companies trying to be profitable by raising subscriptions once in the 20 years they've been running, but instead they are treated like they burned a church. It truly is an elitist system.
How would the car industry change if someone made a 3D printer that could make any part, including custom parts, with just electricity and air? It is a sea change to manufacturers and distributors, but there would still be a need for mechanics and engineers to specify the correct parts, in the correct order, and use the parts to good purpose.
It is easy to imagine that the inventor of such a technology would probably start talking about printing entire cars - and if you don't think about it, it makes sense. But if you think about it, there are problems. Making the component of a solution is quite different than composing a solution. LLMs exist in the same conditions. Being able to generate code/text/images is of no use to someone who doesn't know what to do with it. I also think this limitation is a practical, tacit solution to the alignment problem.