We've gone through this with chess, and the days when a human can do anything to help AI there are long past; the human can only drag it down. It seems to me to be a form of the bitter lesson. I expect this will happen in every field where we don't add guardrails to require humans in the network. I have a hard time seeing regulation happening around this in the US but vanity may also serve, will CEOs be satisfied ruling armies of virtual assistants, or do they need big buildings full of human thralls?

Well, maybe. But most fields of human endeavor aren't very much like a board game with clear, unchanging rules and a single, unambiguous objective.

Have we really though?

This is not a silly question, because it's kind of silly:

Has an AI ever actually gone through all of the steps necessary to become a chess master? Go to competitions, raise in the ranks, take a plane to the masters, organize support and all the real life details necessary? Playing chess is just the small formal part here, and all of the obvious things humans obviously do, well...

I don't agree.

Chess is a perfect information game with objective, easily measurable outcomes.

Show me a software business that looks like that and I'll agree with your hypothesis.

Otherwise I don't believe your metaphor proves much of anything.

Another thing is, I have been playing Chinese chess a bit lately and, yes, Pikafish is stronger than any human player, but if you actually share a board state or list of moves with an LLM they will fail to accurately interpret the state of the board, let alone give good advice on what moves to make. Doesn't really prove or disprove the claim that LLMs will replace human programmers but it does show that the word "AI" is papering over a lot of specifics of particular technologies.

Several influential people are deluded into thinking that LLMs will bring forth AGI. A lot of people don't really understand that LLMs are just one type of AI. This creates a paradigm where we're nearly all-in on LLMs and very few people are thinking of the other ways to engineer task-specific artificial intelligence. Stockfish is a great example of a non-LLM AI that is the best at it's task.

> the days when a human can do anything to help AI there are long past; the human can only drag it down.

I disagree, currently a human guiding an AI agent is far better (more productive for less money) than an AI agent with very loose non-technical guidance and The Ralph Wiggum Loop.

I don't know how much longer humans have, but I don't think they can be cut out of the loop entirely just yet.

In chess? That ended in ~2017.

Sorry, I thought we were talking about Human + LLM agent for software development. I misinterpreted your initial statement.

At some point the whims of the CEOs don’t win. The shareholders demand more growth.

Demand from whom, if not the CEO?

Of course the demand will be directed at the CEO and they will have to provide results or be replaced.

The board, presumably

> We've gone through this with chess

Please never compare perfect information games (https://en.wikipedia.org/wiki/Perfect_information) with actual reality, which is probably many orders of magnitude more complex.

Kind of similar story with math (pure) and other sciences (impure): https://xkcd.com/435/

OpenAI & co keep presenting their achievements at maths competitions as their path to AGI/ASI/singularity. But if you look at that XKCD graph, the way the joke reads is in terms of arrogance (scientists working on purer fields are more arrogant), but the actual science complexity work the exactly the other way around. Impure sciences are orders of magnitude more complex.

Implementing a sorting function = coding (relatively pure).

Creating an entire commercial software product/service with high performance and other professional-level characteristics = software engineering (wildly impure; after 60+ years we can't even agree on the best programming language to use).

Yes, but aren't you just restating the premise in more words?

>We are currently in the "centaur" phase where a human-AI combination

It's probably more accurate to call it the slop phase.