You are aware that more accidents happen per mile at night? If people drove at night just fine, this wouldn't be the case.

That just means we're successful 99.998% of the time instead of 99.999% of the time. It's not a particularly significant difference.

It's significant enough to directly prove that degraded vision leads to worse driving. This is very significant if we decide whether a driving system should only use vision, which can degrade.