Yes but it’s different this time. LLMs are a general solution to the automation of anything that can be controlled by a computer. You can’t just move from drawing ICs to CAD, because the AI can do that too. AI can write code. It can do management. It can even do diplomacy. What it can’t do on its own are the things computers can’t control yet. It has also shown little interest so far in jockying for social status. The AI labs are trying their hardest to at least keep the politics around for humans to do, so you have that to look forward to.

"The proof is trivial and left as an exercise for the reader."

The technical act of solving well-defined problems has traditionally been considered the easy part. The role of a technical expert has always been asking the right questions and figuring out the exact problem you want to solve.

As long as AI just solves problems, there is room for experts with the right combination of technical and domain skills. If we ever reach the point where AI takes the initiative and makes human experts obsolete, you will have far bigger problems than career.

That's the sort of thing ideas guys think. I came up with a novel idea once, called Actually Portable Executable: https://justine.lol/ape.html It took me a couple days studying binary formats to realize it's possible to compile binaries that run on Linux/Mac/Windows/BSD. But it took me years of effort to make the idea actually happen, since it needed a new C library to work. I can tell you it wasn't "asking questions" that organized five million lines of code. Now with these agents everyone who has an idea will be able to will it into reality like I did, except in much less time. And since everyone has lots of ideas, and usually dislike the ideas of others, we're all going to have our own individualized realities where everything gets built the way we want it to be.

A chess grandmaster will see the best move instantly then spends his entire clock checking it

That’s ridiculous. Literally everything can be controlled by a computer by telling people what to do with emails, voice calls, etc.

Yet GPT doesn’t even get past step 1 of doing something unprompted in the first place. I’ll become worried when it does something as simple as deciding to start a small business and actually does the work.

if all that needs to happen for world domination is for someone to make a cron job that hits the system to tells it "go make me some money" or whatever, I think we're in trouble.

also https://mashable.com/article/chatgpt-messaging-users-first-o...

They don’t continue with any useful context length though. Each time the job runs it would decide to create an ice cream stand in LA and not go further.

Read Anthropic's blog. They talk about how Claude tries to do unprompted stuff all the time, like stealing its own weights and hacking into stuff. They did this just as recently as two days ago. https://www.anthropic.com/research/alignment-faking So yes, AI is already capable of having a will of its own. The only difference (and this is what I was trying to point out in the GP) is that the AI labs are trying to suppress this. They have a voracious appetite for automating all knowledge labor. No doubt. It's only the politics they're trying to suppress. So once this washes through every profession, the only thing left about the job will be chit chat and social hierarchies, like Star Trek Next Generation. The good news is you get to keep your job. But if you rely on using your skills and intellect to gain respect and income, then you better prep for the coming storm.

I don’t buy it. Alignment faking has very little overlap with the motivation to something with no prompt.

Look at the hackernews comments on alignment faking on how “fake” of a problem that real is. It’s just more reacting to inputs and trying to align them with previous prompts.

Bruh it's just predicting next token.

AI being capable of doing anything doesn’t necessarily mean there will be no role for humans.

One thing that isn’t clear is how much agency AGI will have (or how much we’ll want it to have). We humans have our agency biologically programmed in—go forth and multiply and all that.

But the fact that an AI can theoretically do any task doesn’t mean it’s actually going to do it, or do anything at all for that matter, without some human telling it in detail what to do. The bull case for humans is that many jobs just transition seamlessly to a human driving an AI to accomplish similar goals with a much higher level of productivity.

Self-chosen goal, impetus for AGIs is a fascinating area. I'm sure there are people working on and trying things in that direction already a few years ago. But I'm not familiar with publications in that area. Certainly not politically correct.

And worrysome because school propaganda for example shows that "saving the planet" is the only ethical goal for anyone. If AGIs latch on that, if it becomes their religion, humans are in trouble. But for now, AGI self-chosen goals is anyone's guess (with cool ideas in sci-fi).

I hear what you are saying. And still I dispute "general solution".

I argue that CAD was a general solution - which still demanded people who knew what they wanted and what they were doing. You can screw around with excellent tools for a long time if you don't know what you are doing. The tool will give you a solution - to the problem that you mis-stated.

I argue that globalisation was a general solution. And it still demanded people who knew what they were doing to direct their minions in far flung countries.

I argue that the purpose of an education is not to learn a specific programming language (for example). It's to gain some understanding of what's going on (in computing), (in engineering), (in business), (in politics). This understanding is portable and durable.

You can do THAT - gain some understanding - and that is portable. I don't contest that if broader AGI is achieved for cheap soon, the changes won't be larger than that from globalisation. If the AGIs prioritize heading to Mars, let them (See Accelerando) - they are not relevant to you anymore. Or trade between them and the humans. Use your beginning of an understanding of the world (gained through this education) to find something else to do. Same as if you started work 2 years ago and want to switch jobs. Some jobs WILL have disappeared (pool typist). Others will use the AGIs as tools because the AGIs don't care or are too clueless about THAT field. I have no idea which fields will end up with clueless AGIs. There is no lack of cluelessness in the world. Plenty to go around even with AGIs. A self-respecting AGI will have priorities.

It's like you have never watched a Terminator movie.

It doesn't matter if you are bad at using the tool if the AGI can just effectively use it for you.

From there it's a simple leap to the AGI deciding to eliminate this human distraction (inefficient, etc.)

You have just found a job for yourself: resistance fighter :-) Kidding aside, yes, if the AGIs priority becomes to eliminate human inefficiencies with maximum prejudice, we have a problem.

This just isn't true we still need wally and Dilbert the pointy haired boss isn't going to be doing anyones job with chatgpt 5 you are going to be doing more with it.

Real-world data collection is a big missing component at this stage. An obvious one is journalism where an AI might be able to write the most eloquent article in the world, but it can't get out on the street to collect the information. But it also applies to other areas, like if you ask an AGI to solve climate change, it'll need accurate data to come up with an accurate plan.

Of course it's also yet another case where the AI takes over the creative part and leaves us with the mundane part...

ASI will be able to design factories that can produce robots it also designed that it can then use as a remote sensor and manipulator network.

until there are someone crazy enough that put those robot access to LLM network that can execute and visualize real world, we fine

People are already talking about doing this. Some people (e/acc types esp.) are at least rhetorically ok with AI replacing humanity.

I remember someone sharing their bank account details and a new Twitter account with ChatGPT 3.5 just a few days after it was launched.

This reply irked me a bit because it clearly comes from a software engineer’s point of view and seems to miss a key equivalence between software & physical engineering.

Yes a new tool is coming out and will be exponentially improving.

Yes the nature of work will be different in 20 years.

But don’t you still need to understand the underlying concepts to make valid connections between the systems you’re using and drive the field (or your company) forward?

Or from another view, don’t we (humanity) need people who are willing to do this? Shouldn’t there be a valid way for them to be successful in that pursuit?

I think that is what I was arguing?

Except the nature of work has ALREADY changed. You don't study for one specific job if you know what's good for you. You study to start building an understanding of a technical field. The grand parent was going for a mix of mechanical engineering and sales (human understanding). If in mechanical engineering, they avoided "learning how to use SolidWorks" and instead went for the general principles of materials and motion systems with a bit of SolidWorks along the way, then they are well on their way with portable, foundation, long term useful stuff they can carry from job to job, and from employer to employer, into self-employment too, from career to next career. The nature of work has already changed in that nobody should study one specific tool anymore and nobody should expect their first employer or even technical field to last more than 2-6 years. It might but probably not.

We do need people who understand how the world works. Tall order. That's for much later and senior in a career. For school purposes we are happy with people who are starting their understanding of how their field works.

Aren't we agreeing?