Not exactly. Sure, if your email is compromised someone could access the secret link, and ultimately your password. But now, you will know that someone else had access to your password, since the link won't work for you anymore. This is crucial information.

Btw. You can encrypt your secret with an optional password (which may be shared via a different channel).