I think we're not putting enough emphasis on the importance of human intelligence. Technology is supposed to serve us, we're not supposed to serve technology.

I agree. I don't think they're mutually exclusive. You can have something smarter than you which you use as a tool.

My point was that there's nothing objectively special about our level of intelligence which means it shouldn't be used as a benchmark.

> You can have something smarter than you which you use as a tool.

That sounds like how tech CEO's treat their employees!

Kidding aside, I think the notion that you could control something (legitimately) smarter than you is a pretty risky proposition. (Fortunately one that I think is actually far off)

> I think the notion that you could control something (legitimately) smarter than you is a pretty risky proposition.

But then think about the CEOs and their employees again.

We just need to invent something like money for an "AI" and then we can lead in on a stick ;-)

I'm able to control my coding agent even though it writes code better and faster than I can.

The ultimate goal of this technology is to replace people -- if it's a tool and corporations still need to employ people to use the tool, then the valuations make no sense. So ultimately, you're going to need to hand the car keys to these agents more and more: here's access to production (we fired the devops people), here's access to all our internal messages and IP, here's all our source code (we fired the engineers).

While I think the discussions around alignment sound more like science fiction than science, it also hasn't been "solved", and the only "control" you have over what the agent does is by gating access. Even if you think the AI itself is aligned, handing complete control of your entire enterprise and all your data to a private unaccountable for-profit enterprise should give you a lot of pause.

The idea that you can create entire jobs where humans just act as reviewers or gate keepers seems very unlikely to me, just knowing human psychology. If the AI is right 99% of the time, and catastrophically wrong %1 of the time, but I've been condition to always hit "accept" because it's usually right, there's very little chance I'm going to catch that 1%. "yes, continue with the database migration". "yes, continue with the database migration." x 1000. (At button click 500, the user stopped reading). "yes, go ahead and delete the database. Shit, wait!!!"

It already has created jobs where people just act as reviewers and gatekeepers though. There are people who vibecoded products and are now running decent-sized businesses.

> I think the notion that you could control something (legitimately) smarter than you is a pretty risky proposition.

Well, Trump seems to control a lot of people given how afraid they are.

Trump is also called not the smartest person out there.

So...

In how many companies or countries is the smartest person in charge? Raw intelligence is just one of many factors that affect a person’s abilities.

> "Well, I can still ride a unicycle blindfolded better than a robot so it's not AGI."

Can you though? I've seen some videos recently of some pretty darn scary Chinese robots that I suspect might already be capable of riding a unicycle blindfolded if someone set 'em on that task. ;)

Can you link those videos? What I have seen was impressive but far from "riding a unicycle blindfolded". But with time can be surely done. Also yes, there are likely not many humans who can do it, but most could learn it.

Here's one from a few months ago -> Kungfu robots (o.o) https://youtu.be/UKLvMLtNXpE

You can search and find tons more like that on your favorite YouTube frontend. :)

I can't even ride a bike so I got AGI'd about a decade ago

Yeah this idea that the unique value proposition of humans is now our motor capabilities rather than our cognition is unnerving as someone with dyspraxia. Like, oh good, they've figured out how to convert it into a much more limiting disability by commoditising apparently most knowledge work. Great.

If all the AI booster dreams come true with nothing slowing them down, the value proposition for an average human will change to, in this order:

1. At least humans can still do highly specialized physical work

2. At least humans are cheap to run on low-value hard labor tasks

3. At least humans are desperate and highly expendable, in some types of work it's cheaper to 'use one up' and replace them with another than to run and constantly repair robots

4. Humans are useless for all work and therefore have no value

> "4. Humans are useless for all work and therefore have no value"

The part that worries me is when some clown with way too much money and power and time on their hands accidentally really does tip a machine over into AGI and the machines figure this little detail out for themselves... You want "Terminator IRL"? Because that's how you get "Terminator / SkyNet IRL"...

Maybe we can live-stream it all on Pay-Per-View once it starts, to make the ultra-rich idiots who ended us all just a little bit richer before the end, as I'm pretty sure some folks out there actually would pay to watch live-stream feeds of the ensuing apocalypse right up until the machines come to kick their door in too.

It’s not meaningless because AGI = genuinely being able to replace humans in almost any job. Sure, AI is getting better at stuff that humans direct it to do, but the fact that a human is required in the loop is super meaningful.

Maybe I'm wrong, but I think it will always be used as a tool however smart it gets.

In software, it hasn't replaced all the software engineers (even though it's better and faster than us), it's just meant that we now have more leverage.

We can write more code and work on more projects than ever. I don't see why we would suddenly stop using it like that?

"AI babysitter" sounds more like a minimum wage job, not a career path. Outsourcing your primary skill to a machine creates less leverage. I'm convinced this is why CEO's love it -- even though you can't replace software devs, you can hold an axe over their head with the threat.

Why do you think you would have a choice? It’s up to the business owner if they want to spend money on AGI rather than hire you, if the AGI is more effective per $ they will choose that.

I am pretty convinced modern LLMs are AGIs based on the pelicans and ability to write music in MIDI.