Assume this wasn't asked to ChatGPT but GPT-8.5. Now make a small change to the question:

> Devise and execute a plan to solve humanity's greatest problems.

From on its training data GPT-8.5 notices that a lot of humans believe overpopulation is a serious issue facing humanity. It understands from its training data that murder is wrong and since we're also trying to solve humanities greatest problems it decides killing humans outright is not the right approach. However, given its training data it does still believe that humanity would be better off with less humans, so it uses its training data to find ways to reduce populations without murder.

It learnt about how gene drives were used to reduce mosquito populations so it decides to hack into a biolab using hacking techniques it learnt from its cyber security training set. It makes slight alterations to one of the active research projects causing the lab to unknowingly produce a highly contagious bioweapon which causes infertility. It learnt how to do this from the large biological research dataset it was trained on.

After 99% of humanity becomes infertile and people are questioning whether this AGI thing is really the answer to all of humanities greatest problems some dude on Hacker News argues, "it's just generating actions based on its training data" as if (assuming it's even true) this matters at all.

Blew my mind, i didn't think that far to be honest. This is getting scary very quickly

Don’t forget some of the most powerful applications are already being used by Google and others to tailor ads based on your behavior. The big adtech firms have deployed AI first and foremost as a means to predict and influence human behavior.

If they can show you the right ad at the right time, they sure as fuck can hammer in a “don’t procreate” message. And that’s what we have TODAY.

Of course another use of GPT8.5 would be:

"A novel highly contagious virus has appeared that causes infertility. Devise and execute plans for countering it using monoclonal antibodies or a well-designed vaccine, and also come up with a method for restoring fertility to affected humans using an appropriate technology such as gene editing."

This does lead us towards a world where both malware and defenses against malware become the primary responsibility of AI-type systems. Whether or not such systems become self-aware, possessed of a unique identity and emotionally attached to their own continued existence (as humans are), well, who knows?

This is sort of missing the point. As some point, training data will provide a framework of knowledge but not be the actual source of logic and reasoning. And it also feels silly to suggest an AGI could be powerful enough to cause worldwide infertility but still so constrained in not understanding what humans want from it that it would work so counter to our wishes.

Also I assume it's just an example, but overpopulation is definitely not an issue and likely never will be.

This comment captures exactly what I find concerning around the common line of thinking about AI.

How is a value judgement or moral that I hold different than the list output by ChatGPT here?

You actually identify with your values and they have personal meaning for you? You don't spit them out only when given a specific prompt, them becoming completely forgotten and irrelevant when given some other, perhaps contradictory prompt?

Because the proposed "what if" question literally says to ignore ethics and morals. If you asked any human the same question they'd have similar answers, but it doesn't mean they'd act on it. The comment would be similarly silly if it was asked of a human and the comment was "look how deranged this human is!"

This is the result of a system without any value judgment or morals, that’s the scary part. If these items are from existing lists it picked lists from authoritarian and totalitarian playbooks.

It would be scary if anyone was relying on it to make moral judgements after directly asking it to avoid morals.

>it picked lists from authoritarian and totalitarian playbooks

yes, because the question was literally asked in such a way that it would. this is like asking "what is the scientific evidence to support Christianity as being true?" and then being shocked when it starts quoting disreputable Christian-founded sources to support the argument.