Same reason people are afraid of getting robbed at night when they're more common during the day when everyone is at work, or why we're scared of heights and planes, but cars and horseback riding don't phase us. Fear is an emotional reaction, not a rational one.