Nassim is arguing that RISK is a separate discipline, separate from the domain where risk applies. That a person building AI is not the correct choice for estimating AI risk.
You don't ask gun making companies to make policies regarding risk of gun owning in society.
Having worked in the medical domain in the past, paramedics and medics that should know better were taking extremely high health-related risks (riding a motor bike => crashing and burning to death, smoking => dying from lung canceer, speeding onto a crossing => dying in an ambulance crash before arriving at the 999 call site etc.).
So risk is indeed its own discipline, separate from the domain where risk applies, even if we are talking about the life-rescuing domain of medicine: a person rescuing another is not automatically an expert at reducing their own (health/life) risk exposure.
While neural network research results are published in NeurIPS, ICLR, ICML, ECML/PKDD, JMLR etc., risk results tend to get published in the risk community at conferences like SRA [1] (Europe: SRA-E) and the likes. I'm not a fan of this academic segregation, merely describing what is going on (in my own teaching, for instance, I include risk/ethics consideration along the way with teaching the technical side, to avoid ignorance caused by over-compartmentalization).
[1] Annual Meeting of the Society of Risk Analysis, https://www.sra.org/events-webinars/annual-meeting/
Simple. Nassim says there are 4 quadrants, one axis Mediocristan-Extremistan, the other Simple-Complex payoff.
Building a bridge is Mediocristan/Simple payoff, a well understood problem with no black swans. So it's easy to compute risk.
Other stuff is Extremistan/Complex payoff - financial trading, pandemics, AI. And he argues that you need RISK professionals for this quadrant, because people working here (traders, virologists, AI builders) do not understand how to compute the risk correctly.
https://www.researchgate.net/profile/Gaute-Bjorklund-Wangen/...
I think this is often fair. It's actually one of my primary criticisms of the NeurIPS/ICML survey.
FWIW, people working on AI Safety -- like, actually working on it, not philosophizing about it -- are some of the most incredulous and annoyed about the "AGI => extinction" crowd.