It doesn't magically abolish your right, but it is evidence against you if a competitor actively and overtly uses your mark and you don't enforce it even to the level of writing to them seeking a rename or a license.

Not only is there a risk of estoppel (probably not if they can't show anything explicit or intentional by you), even other competitors can point to your willful inaction in support of their argument to a judge that your trademark has become generic.

As noted above, I'm not a lawyer or giving specific legal advice here, but I have worked with lawyers in dealing with trademarks.