I think you should read Taleb's followup Antifragile sometime. The point is not that black swans are unlikely, but that they are catastrophic, and we can't even quantify how unlikely they are, so we have to find ways to prepare for them without basing it on probability estimates.

That "something really bad" might be localized ecological collapse, which I find pretty compelling, too.