I sort of agree: You don't automatically lose anything. But once you sue to enforce your trademark, the court looks at the body of evidence - and may determine that the trademark has been diluted and has been lost. There is no AFAIK predefined threshold (or way to measure, for that matter) the dilution of trademark - a different judge/jury may reach different conclusions.

And basically, that's why most trademark lawyers will act on even the smallest unauthorized trademark use - they do not want to leave it to chance.