I've had comments voted lower than -3 (you start at 1) plenty of times. My favorite one is one comment criticizing Microsoft that oscillated from 25 to -10 a few years back (yeah, I refreshed this one quite a lot to see how it changed, that was fun).

But I don't remember anything lower than that recently, so it can be a new change.

5 times, the lowest you can go is -4.