That is true up to a certain point (for instance, in my experience, having bounding boxes that are not pixel-perfect acts as a regularizer), but there is also a good chance that you are mislabelling edge cases, situations that happen rarely, and that definitely hurts the performance of the neural network to make a correct prediction on these difficult / uncommon scenarios.

We did some interesting experiments with Go where we inverted the label of who won and measured what impact that had on the final model. This is a binary label so it's probably more impactful (it's the only signal we are measuring)

From memory it had only a small impact (2% strength) with ~7% of results flipped, at 4% it was hard to measure the impact (<1%)

Also, this is applies to mislabeled data in your training set, right? Not a good thing if it is in your test set.