> seems to not have much awareness about what the consequences of such a project might mean

That's the worst part of it.

Is the current generation of LLMs dangerous? Probably not. Will the next generation of LLMs be dangerous? Maybe not, if we're lucky. The one after that? Nobody knows.

But projects like this don't so much rely on there being no danger, but rather, the question seems to not even have been considered. And that's crazy, because the assumption that things are harmless is getting more and more shaky the more capable these models become.

I can kind of understand it. After over a decade of spending time in the influence zone of Bostrom, Eliezer, and LW crowd[0], I find myself torn. Half of me is acutely aware of the danger potential of what we're discussing. The other half wants to play with chaining the current LLMs, text-to-$X and $X-to-text models, and domain-specific classical software in hopes of bootstrapping a proto-AGI, because how cool it would be to pull it off.

Good my ADHD makes me write HN comments on LLM threads instead of working on the latter, because I'm not sure I could stop myself otherwise. But this also means I can very much imagine only thinking about the cool parts, and not considering the dangerous ones - because that's how I feel about 90% of the time when thinking about current SOTA models. It's just too damn exciting.

(Hell, it's the first thing in tech in the last decade, that I find exciting.)

--

[0] - Whatever one may think of them, I find it sadly ironic how Eliezer and his disciples were considered pseudo-scientific doomsday AI cultists a decade ago, back when the whole thing was only a far-future concern, how they're still considered pseudo-scientific doomsday AI cultists today, after AI field made a series of rapid jumps, landing us exactly in the scope of things they were harping about all that time - and how I bet they'll still be considered pseudo-scientific doomsday AI cultists after someone unleashes an unaligned AGI and ends us.

To be honest with you, sometimes things seem so shitty on a world level that I find myself entertaining the "if AGI is possible then let's risk the coin flip of extinction or greateness, I'm tired of all this status-quo bs"

Can't help but think people who want to dabble in this stuff should honestly be doing it on the moon or mars or something where I think a fair bit less can go wrong.

I know it's technically crazy sounding, but for all we know, Earth is the only place with biological life, I think it's better to flip the coin somewhere else if that's what you want to do. Don't screw up here.

Yes, maybe the AI will come for Earth, but I feel like there's less chance of shit going really bad with some distance.

This is indeed our very best shot at taking down global capitalism

Of course, doing it wrong is also the best shot to ensure capitalism eats the universe.

Your point about the doomsday AI cultists is nonsense.

The doomsday AI cultists didn't come up with any of these ideas. The Matrix was a hugely popular movie in 1999.

chatGPT doesn't make the person who has believed since 1999 that AI is going to turn us all into Duracell batteries more correct. It is the same basic thought with with paper clips replacing batteries.

Their point is that AI is now suddenly much closer to AGI than any experts predicted it would be. Well other than the “Technological Singularity” guy, who was ludicrously on the money - I think he said 20-30 years from 1993?

>consumption by just anyone.

ah yes, the only people consuming such things should be The Big Corporations and Big Governments?

---

Snark aside (Sorry, I'm just a bit tired of people having a kneejerk reaction to this without even thinking),

we're talking about autonomous agents. Huginn and the like have existed for years. If they can be even somewhat better automated with NLP and the sort, it'd be excellent instead of having to actually manage and configure/code individual agents. and Guess what, these agents have ALREADY been connected to the internet. For years.