Yes, I think you have this exactly right. We've dodged the bullet for a few hundred years, which our brains are predisposed to think of as a long time. Then we (well, not all of us, but enough of us to matter) (mis-)apply inductive reasoning and conclude that because doomsday predictions have always been wrong in the past they will continue to be wrong in the future.

In fact, you can sum up the problem as a pithy slogan: Doomsday predictions are always wrong. Until they aren't.

Someone should make T-shirts.