It would have to be trained in 100% of all potential scenarios. Any scenario that happens for which they're not trained equals certain disaster, unlike a human who can adapt and improvise based on things AI does not have; feelings, emotions, creativity.

You're still operating with the assumption the AI is doing independent work, it is not, it is advising the people doing the work. That is why people are the ones be augmented and enhanced, and not the other way around: people have the capacity to handle unforeseen scenarios, and with AI as a strategy advisor they'll do so with more confidence.

I have witnessed no evidence that would support this claim. The only contribution of LLMs to mathematics is in being useful to Terry Tao: they're not capable of solving novel orbital mechanics problems (except through brute-force search, constrained sufficiently that you could chuck a uniform distribution in and get similar outputs). That's before you get into any of the engineering problems.

https://deepmind.google/discover/blog/funsearch-making-new-d... seems to be a way. The LLM is the creative side, coming up with ideas-and in which a case the “mutation’ caused by hallucinations may be useful. Combined with an evaluation evaluator to protect against the bad outputs.

Pretty close to the idea of human brainstorming and has worked. Could it do orbital math? Maybe not today but the approach seems as feasible as the work Mattingly did for Apollo 13.

And the LLM can be replaced by a more suitable search algorithm, thus reducing the compute requirements and improving the results.

You do not have them solving such problems, but you do have them in the conversation as the human experts knowledgeable in that area work to solve the problem. This is not the LLM AIs doing independent work, this is them interactively working with the human person that is capable of solving that problem, it is their career, and the AI just makes them better at it, but not by doing their work, but by advising them as they work.

But they aren't useful for that. Terry Tao uses them to improve his ability to use poorly-documented boilerplatey things like Lean and matplotlib, but receiving advice from them‽ Frankly, if a chatbot is giving you much better advice than a rubber duck, you're either a Jack-of-all-Trades (in which case, I'd recommend better tools) or a https://ploum.net/2024-12-23-julius-en.html Julius (in which case, I'd recommend staying away from anything important).

I recommend reading his interview with Matteo Wong, where he proposes the opposite: https://www.theatlantic.com/technology/archive/2024/10/teren...

> With o1, you can kind of do this. I gave it a problem I knew how to solve, and I tried to guide the model. First I gave it a hint, and it ignored the hint and did something else, which didn’t work. When I explained this, it apologized and said, “Okay, I’ll do it your way.” And then it carried out my instructions reasonably well, and then it got stuck again, and I had to correct it again. The model never figured out the most clever steps. It could do all the routine things, but it was very unimaginative.

I agree with his overall vision, but transformer-based chatbots will not be the AI algorithm that supports it. Highly-automated proof assistants like Isabelle's Sledgehammer are closer (and even those are really, really crude, compared to what we could have).

No