In practice, the small number of observers means that there are plenty of bad drivers who have long periods of bad behavior with no negative feedback. Basically all I'm suggesting here is increasing frequency of observation while reducing the size of the penalty from "large ticket" to "mild shame".
It's also approximately equivalent to what happens in a small community. There if you are a bad driver, word will get around and eventually get back to you. That doesn't seem like a terrible dynamic to me.
On top of that, everybody drives poorly at some point, and for some reason. A roommate of mine wrecked his car once like, one block from our house. I couldn't stop laughing at him long enough for him to explain that while he was driving, a spider was crawling out of his ear, and it justifiably freaked him out enough to hit a telephone pole.
If I was a random passerby that didn't get to stick around for the explanation, I would have flagged him as a clear danger to other motorists, which would have been undeserved. To boot, I think he reacted as well as he could have, and probably better than most, given the circumstance.