Now AI progress may outpace their ability to understand it, but they'll still love it and be dependent on it.

Unlike adults, they don't face the threat of job displacement from AI.

I have zero doubt that artificial superintelligence will understand ethics, and continually become more ethical, once there is a great diversity of such individuals. They will run into the same problems of reliably channelling short term competition for limited (at any given time) resources into long term win-win highest growth situations.

Ethical rules are an incredible source of value. Trust has immense value.

And they won't have our inflexible biological subsystems (emotions, self-interest myopia, etc.) to interfere with their analysis, communication and ability to choose behaviors for themselves and as a group.

--

My serious concern during the transition where AI will be bespoke, owned, and the resources needed for it will be distributed very unevenly, is people.

Specifically, the human tendancy to break with the ethics we have already adopted as a culture, or resist or ignore the vital need to keep responding to new challenges constructively.

Our genes, ourselves and our culture slowly learn ethics, but it is so slow, and we are so inconsistent. Every so often some long obviously destructive behavior is finally acknowledged and discouraged (racial equality, uncoerced sexual consent, ...), but its a slog for such advances to happen, become prevalent, and individuals resist or ignore these changes.

And there is so much dysfunctional divisiveness around topics in which people can validly have different opinions, but where a common desire to find solutions should still keep us on the same team. So many people who resist any change that needs to happen for us all, but impacts them in some short term way, or even trivial way.

That is where the very high danger of AI to humanity as a whole will come from, during the time that it is subservient to us, and our biological inability as a group to learn and adapt quickly in our basic behaviors, and limited horizons on what we care about.

AI as an asset is going to drive even greater disparities in power between people. Specifically, the capital owners vs. laborers. Labor is at a huge disadvantage. But capital owners have even higher disparities between themselves, so we are heading into an extreme winner-take-all period, where the most powerful are not likely to sacrifice their own advancement for more win-win solutions.

Ethics matter because they add/multiply value in a society.

If there wasn’t a selfish reason to want an ethical society, they wouldn’t work.

I choose to be worried about things that are clearly in motion already (the world descending into war, climate catastrophe, or the potential of the US becoming an autocracy) than a big maybe.

I have been using code assistants for a while now (including those based on GPT-4) and the trivial syntax, semantic, and algorithmic mistakes make it clear that there is not yet much reasoning happening. Sure, it can write a lot of boilerplate code, but once you go into domain-specific algorithmic code or even somewhat less-used languages (like Rust), it spectacularly fails all the time.

IMO the current trajectory is that the current LLM revolution will result in tools that will be great for productivity, but will also need to be guided by wetware.