It's about very good general problem solving software that's way beyond the capabilities of humans while not being aligned with human interests. Not because the software is evil, but because aligning values is an unsolved problem (humans aren't even totally aligned - and values also change).
If you have an intelligence that's very good at achieving its goal and you don't have a good way to align its goal with human goals, you can very quickly get into trouble if that intelligence thinks much faster than you do.
https://www.lesswrong.com/posts/mMBTPTjRbsrqbSkZE/sorting-pe...
https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden...
https://www.lesswrong.com/posts/BEtzRE2M5m9YEAQpX/there-s-no...
https://www.lesswrong.com/posts/5wMcKNAwB6X4mp9og/that-alien...
Also, self-driving cars were mostly hyped up by companies, FSD is quite obviously a hard problem, much closer to general intelligence than the average NN application.
When I was a kid, I remember wondering how Soviets were obsessed with faking photos. A few years later, I saw Terminator 2 and realized that faking videos was also a thing. The tools for it would clearly get better and better over time. When I studied ML in the early 2000s, it seemed obvious that pattern recognition tasks such as image manipulation would be "easy" for computers, once we found the right approach and made the ML systems big enough. In the end, I decided not to pursue ML, because jobs were still scarce and I found discrete problems more interesting. That was probably the worst career mistake I've ever made.
Or, watch the latest Veritasium
#2 see linked video, he sits in the backseat, there is no driver and no one to take control.
Few have predicted a reasonably-capable text-writing engine or automatic video face replacement, but many have predicted self-driving cars would have been readily available to consumers by now and semi-intelligent helper-robots being around.
Just because unforeseen advancements have been made, does not mean that foreseen advancements come true.