An annoyance, yes it is. But I think it's a good feature. Not making mistakes is the most important thing to improve your touch typing speed. If this annoying behavior makes you slow down and stop making mistakes, instead of using backspace without thinking about it, then it's well worth it. That backspace reflex comes very quickly anyway, it's not really something you need to train.