When you find a bug, by all means, go ahead and fix it. (Well, not always. You need to make sure your fix is not likely to introduce a more severe bug. Not all bugs made equal and some guts-feeling required).

If your goal is to make your system as robust as engineerically possible, you should most likely pay more attention in some directions and less in other. It's not only about quantity - you better have no critical bugs then have only-a-very-few critical ones.