Lots of older buildings have asbestos that's preventing fires and won't cause noticeable harm so long as it's undisturbed. The fibres are a problem, but a sheet of material under a layer of plaster isn't breaking off fibres that can get into your lungs. If your job is to rip out plaster and re-model buildings then that's a huge problem, but if you just live in a building you're getting much more benefit from it not catching fire than you'd get from very minutely reducing your risk of lung damage from asbestos.
In fact even for a brand new building, although it's banned in some places, "Let's use asbestos" makes more sense than "Eh, so what if it burns down and a few dozen people die", so if those are the available choices you should go with asbestos.
So it is with the available COVID-19 vaccines. SARS-CoV-2 is very dangerous and, outside of New Zealand and a handful of Pacific islands, widespread, so you're just way more likely to get very sick or die as a result of not being vaccinated than suffer a serious side effect, known or hypothetical.
What if, knowing what we do now about it, OSHA required asbestos-lined jackets?