This type of research requires experimentation (mostly failures) on extremely complex real-world equipment. Same with the nuclear weapons. AI being able to magically figure it out without experimental grounding is pure and absolute fantasy, used by companies like OpenAI and Anthropic as a justification for monopolizing AI R&D. In a sense it's not surprising this idea comes from rationalism-adjacent folks, as rationalism is mostly about the idea that experimentation is irrelevant and you can infer anything using just logic alone.

Okay, I’m inclined to agree there.. but I can’t find the reference now, I also read that the worry is that the complexity required is coming down fast, and while maybe it’s not going to happen in a basement , there could be a small scale lab, not subject to rigorous certifications and checks that offers crisper as a service, and ai could be used to .. just perturb a protein a little bit so it doesn’t trigger some known virus black list, and people could just be ordering things online.

> In a sense it's not surprising this idea comes from rationalism-adjacent folks, as rationalism is mostly about the idea that experimentation is irrelevant and you can infer anything using just logic alone.

Yeah IIRC Yudkowski famously said something about a super intelligence could derive the theory of gravity correctly by seeing only three frames of a video depicting an apple falling from a tree. This is the same Less Wrong nonsense, rejecting how vital and irreplaceable experimentation is.

There's an infinite number of explanations for the location of an object in three equally time-spaced instances. Not to mention limitations of the measuring equipment itself.

There has been a lot of research into discovering new physics (starting by discovering old physics) since the last 5-6 years and it always require:

- A lot of high-quality data - Some careful design - (Not always) some external knowledge to guide the solutions

And this is using specialized NNs for physics, where you often know underlying equations. Kind of crazy that some people are so delusional about that.

>rationalism is mostly about the idea that experimentation is irrelevant and you can infer anything using just logic alone.

Thanks for putting it the way you did. I didn't knew it was meant be that way, but it sort of confirms my suspicion that people who use the term 'rational' and 'logic' loosely often to dismiss an opposing view never really seek experimental results before having a point of view.