That's why it should not be HTTP GET endpoint. Get endpoint should only be when request is idempotent. Use HTTP POST for your usecase.

Is it possible to embed a link that uses POST in an email? I can't think of a way unless form tags work, but then the link wouldn't work in a plain text email reader

You’d need to send the user to the verify page and populate a form with their token from the url. Then submit the form, either automatically or by getting the user to manually hit a button.

You should not submit the form automatically. Tools like Microsoft O365 ATP run any links in an emulated browser with Javascript support. These will, in some cases, happily autosubmit the form for you.

We've done this for over 2 years now, and over 200k users never ran into this. Also not within government users.

It could very well be that your specific Javascript does not run automatically or does not run correctly. I see the same with one of our auto-submitting forms. I do not know whether or not that is intentional on Microsoft's part. But other users have had different experiencies, so be aware that Microsoft may 'fix' their issue on some day and all of a sudden all your users will start clicking/unsubscribing/whatevering automatically.

See https://blog.healthchecks.io/2019/12/preventing-office-365-a... for someone who did have this experience.

Turn on O365 “link scanning for malware” and Salesforce onetime links for password resets etc stop working.

How do you know your users have never run into this?

People don't tend to report problems like "my account was activated sooner than I expected"

We use it to log in users. We quickly get complaints if something doesn't work well.

But your users ought to believe it works well to have google's servers respond to the activation link instead of requiring them to click it themselves, so no complaints.

The verification process serves only you, the administrator. To everyone else it's a tedious obstacle.

Nobody will reach out to you to say "I made it through the registration process just fine but it was slightly less burdensome than I expected, is everything OK?"

If you want to know if Google is hitting your activation URLs, check your access logs. Your users will almost certainly not realize it happened. Even if they do notice it, there is no impact on them and no motivation to inform you. You would have to be extremely lucky to hear about it from a user.

Half of unsubscribe links seem to auto-submit. Are they all broken in O365?

This is how mail chimp does it, I believe. JavaScript to submit the form automatically.

Depending on how your app works, non-idempotent links in emails can often be an over-looked csrf vector. Sometimes people also make such links auto log people in which can be problematic.

You can use <form method=POST> in email body (obviously does not work in plaintext mode)

I this sometime triggers a warning to the user (something like “Are you sure you want to submit form data to external site?”), which may not be the best end user experience.

If you put a header “account verification form” above the button, it's a better experience; the user knows what the computer is calling a form.

That sounds very phishy.

You're assuming the email is HTML. No, not all users use HTML email.

What do you think the second sentence of their comment is talking about?

This endpoint is idempotent - clicking that link multiple times has the same effect as doing it once.

Correct. Idempotency isn't precisely the right concept to appeal to here. The right concept is that GET requests are assumed by convention to be "safe," which implies they aren't tied to user interaction. "...user did not request the side-effects, so therefore cannot be held accountable for them" (https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#:~:te....).

Presumably there is a different content for the first response when the token is still valid, otherwise this would be a pointless link.

Not if it "disappears after access"

I was just wondering if a web page that counts visitors is idempotent. Not?

It changes state on the server, i.e. the counter. So no.

I’d say it depends on what the counter is for. If it’s used to bill the user per pageview, then it’s user-visibly not an idempotent action, so no. If it’s used to estimate site speed, then the user doesn’t care, so yes. (In fact, analytics is the example I see most often under “non-idempotent stuff it’s OK to do on a GET”.) If it’s used to display a counter in the site footer, then you might wish the user cared for your bragging, but they probably don’t, so yes with a disapproving glance in your general direction.

Well, this is why your email provider should not open your links for you. Use a different email provider instead.

I think I'm going to continue using the one that is pre-opening links that should be idempotent so that it can check them against its heuristics for spam or phishing. That's been really nice to have.

And I'll instead refrain from using sites that inappropriately provide bare get URLs that are really state-mutating booby traps in disguise.

Lol, the nerve of the other reply: "stop using the most widely used email provider in the world". I liked your reply better

I believe this is how they fetch images without meaningfully accessing tracking pixels.

If everything send to gmail is opened upon arrival and cached, you know nothing about when or if the recipient actually opened the email.

Last time I looked into this, Gmail was not loading and caching images. Is there any evidence that this has changed?

What is being described here is likely being done for some other purpose.

Google claimed to do that.

> Instead of serving images directly from their original external host servers, Gmail will now serve all images through Google’s own secure proxy servers.

Https://gmail.googleblog.com/2013/12/images-now-showing.html

Proxy servers, not caching servers.

When are proxy servers not caching?

I have an email from 2015 where I reported this as a potential vulnerability in the Hartl Rails tutorial, having seen it myself back then. Consider "verified" ashley madison accounts in their breach and this scenario.

This is not just a Gmail thing. Most corporate mail filters visit a link and scan for malware as a feature.

Make the user take action after opening the link. Like click a button.

And make sure the action is a POST instead of a GET. GETs should never modify important state.

This is the correct answer. Just because the norm is to embed verification hashes in URLs to be clicked, doesn't mean it's the right way for it to be done.

Why not send a short random code by email for the user to then copy into the sign-up form they were in the process of filling in?

It takes more effort and more users will decide to move elsewhere. I don't really believe that if someone can't bother to copy code from e-mail, he's worthy to have as a client, but some company are obsessed by metrics and percentage of successfully registered users is one of those metrics.

I understand your way of thinking, but we ended up having a flow for a government site where users had 2-3 steps what normally could be done in 1. Also many were not tech savvy and confused. So we ended up adding JS to automate the click.

You automated a click on a government website? So tell me: how'd that audit go?

This depends on your willingness to turn away business, and may not even be legal depending on where you work. In the United States, I would not want to defend that copy and paste scheme as being compliant with the Americans With Disabilities act having seen usability tests from people trying to accomplish that exact workflow using screen readers. Remember that things like cognitive impairments count and, like vision and motor control/range of motion, most of us will be affected at some point in our lives.

What I do think would be reasonable is having a well-labeled link which takes you to a confirmation form: someone can follow it easily and choose to submit it with far less friction and it leaves standard web semantics intact.

Clicking a link (one action) is easier than copying a code and pasting it (two actions). It's possible the user will copy the wrong thing or paste the code into a wrong field, including the browser address bar.

All of that may affect the sign-up rate.

Kinda. I often read my email on my phone while working on my desktop. (Or visa versa). In these situations, a code is always better. I hate the links personally.

How many times having to click a link (instead of entering a code) stopped you from finishing a sign-up process?

It’s pretty easy to measure. I had a site with a verification step. And we would see like 20% drop off of people who clicked on the link but never confirmed. Not sure why. We didn’t have them copy and paste anything, just click a confirm button.

Switching to no confirm obviously changed this to 0% drop off of people who clicked the link, but the number of people who clicked was the same.

It was curious to me why people wouldn’t go through with the confirmation step, but never learned why. We just learned that for some reason more people click once instead of twice.

How would you have known if that 20% were real people and not bot activity?

I don’t necessarily. But they have active accounts that do stuff and had the drop off activity consistent with “normal users.”

So it doesn’t matter to me if they were bots or not.

For example, 100 users clicked on the first link, 80 completed, and had normal account activity (clicking on stuff, uploading and downloading things, etc).

100 users clicked on the second link and then had normal account activity.

Maybe they were all bots, but they seemed human based on the “normal activity.”

Nobody remembers the exact moment they stopped thinking about something because it was easier not to.

Ragequitting is one way to exit a process, but just not going to the next step from distraction is surely more common.

I'm asking because often when I talk to people about things they hate, they end up admitting it's not that big of a deal. The annoyance is minor enough they don't look for alternatives or abandon whatever they were doing.

The original discussion was about clicking links vs reading and entering the code in sign-up confirmations. The former takes less steps and is easier to complete. Power users with unusual habits might disagree. But if they complete the sign-up anyway, it makes more sense to focus on regular users.

> user to then copy into the sign-up form

Extra steps are hard and boring and people don’t want to do them.

I consider myself a savvy user and I want to click a link. Not click a link, then look up a code from the email, then paste, then click submit.

I’d live with having to manually click “I’m sure I want to unsubscribe” or something.

This is most annoying when the site wants me to type in my email address to unsubscribe. I have lots and lots of different email addresses that funnel into a single one. When the site doesn’t put my address in the “To” field, I dont know who they sent to.

Services should be respectful of users time.

>This is most annoying when the site wants me to type in my email address to unsubscribe.

I've become increasingly suspicious of this practice

if my email address is in the URL, why don't you autofill that email box for me? if it's not in the URL, why aren't you fetching it from your database using my unique hash in the URL? do you even keep any records of email subscription preferences? am I just signing up for more shitty spam by giving you my email, again? am I just being marked as 'active' i.e. fresh meat, somewhere in the spammiverse?

these questions become more poignant and the suspicion more fiery when, low and behold, it turns out you are still subscribed

I don't fill them in any more, I just block the sender

don't get me started on the "it may take up to 28 days for our systems to register your desertion" bollocks -- I'm not working a contractual notice period, or running a lap of dishonour. it's a bitflip to 'false' in the 'is_pesterable' column, else a respectful deletion. it takes microseconds, not weeks!

That too as I suspect the unsubscribe links may just be data gathering.

I typically never type in any information because I assume the site doesn’t know and wants to know.

I’d rather just set up a kill rule on my end than risk getting my email on one more list.

There were good suggestions in other comments in this HN post.

One of them mentioned that you can continue keeping things as a 1 click solution with the token in the URL, but instead of doing the destructive action upon visiting the link -- instead you would get sent to a page with a form where the token is put into a hidden field that gets auto-submit as a POST request with Javascript.

This way from your POV it's a 1 click solution. You only waste a second waiting for the redirect and if the user doesn't have Javascript enabled you can <noscript> the field as being an input field which is pre-filled out based on the value from the URL (this can be done server side).

Now everyone is happy, unless gmail is going to go as far as auto-following redirects with JS enabled.

Ideally they wouldn't be following redirects _if they're POST requests_, right?

True, that was a bad choice of words but gmail could still load the page, execute the JS which in turn submits the form which is a POST request to your back-end. It's technically not following a redirect but it's doing things beyond just visiting the URL linked in the email due to it executing JS.

Ah interesting, so you're saying that GMail is not likely to be avoiding the POST requests if they're in the JS code? I have a passing, non-professional familiarity with Web practices, so this isn't something I have a great intuition for.

The basic flow would be:

    - You GET /reset/abc123
    - Your server responds back with a page that has a form
    - There's a hidden field with the token
    - Javascript kicks in and on page load executes the form as a POST request
    - Your server responds to that POST request and does whatever it needs to do

All of that is kicked off by gmail visiting /reset/abc123, and now it comes down to whether or not gmail's pre-visiting code will run the JS on the page. If not, then the above workflow fixes this issue, if it does then you're in the same position as avoiding all of this and having a GET /reset/abc123 perform the destructive action.

Right, it's the second part of the fourth bullet that I was asking about. Basically, that it sounds like it isn't feasible for Google to avoid POST requests if they're being submitted from JS.

Are we really going to continue to break the paradigm that GET requests should be idempotent to save people an extra click or Ctrl+C and Ctrl+V? Standards matter. In this case Google are doing something that should be allowed, but being criticised for it because it breaks badly implemented services.

Entering emailed or texted codes is becoming more common with 2FA for banking, PayPal etc. anyway so I think most people are going to broadly manage.

Sorry, GET requests aren’t idempotent. At the minimum they create log entries. So you can DDoS servers by filling their logs with “idempotent” GETs.

UX is important, and I think saying “suck it users, I’m going to use GET the way I think is write” is not a positive way of thinking about it.

I think the problem is just the mechanics of POST not being allowed in an email, so if there’s a way to POST from just clicking on a link I think we should use it. But there’s not, so having a GET that triggers something is the least bad thing. I like it better than javascript and forms in email. And better than autosubmitting, hidden forms on load.

Verification hashes in URLs are fine, as long as accessing the URL does not invalidate the hash yet.

This is how Steam does it.

Thanks. That's good advice.

Btw you could just have JS do a POST request, the user doesn't need to do anything except open the page. This is how unsubscribe pages work.

Thanks. I don't use JS, just Go with HTML templates. I populate the form now with {{ .code }} from the URL so the user does not have to copy/paste the code. But they do have to click 'Submit' to post the form. I think this is a reasonable approach that most users are OK with.

You could add a single line of JS to have it auto submitted as well

Wouldn't that be equivalent to just doing a GET?

No... Because you have to do a GET, execute JavaScript, and make a POST request. Bots don't execute JavaScript and no well-behaved bots are going to make POST requests

No, because if you curl the original url the POST for the second url won't be triggered, but if you navigate your browser to the url, the browser will trigger the POST.

That presumes everyone executes random JS or has a browser that supports it.

Create a "click here" button, hide it by adding it a class or a style with JS.

There is this thing called an HTML form. Every browser supports it.

It should go without saying that you provide a fallback.

Depending on context and implementation details, this can often be a security issue (csrf or something similar). Probably not in the unsubscribe case though

Obviously you need to make sure your API is not susceptible to CSRF but that goes without saying... Should I also tell him to password protect his database? :P

If you follow what the parent is suggesting (open a page with a get request that has js which does a post request automatically with no user interaction), its probably impossible to not be susceptible to csrf

I'm really not sure why you think that, but that's just not true at all. For starters, if it is a JSON API then a CORS request will be done for all cross-domain API requests. Even a missing CORS configuration would then block the CORS request from third-party domains.

So the situation is: there is some url you open (with a normal get request. Typically but not neccesarily from an email), then that url does the non-idempotent POST request without any futher user interaction.

A malicious page that knows the url used in the email could open the url from the email in a popup. The js will execute in the popup, and do the POST request. It doesn't matter how much csrf protection you have on the POST step, if anyone can trigger it with no user interaction just by opening some page with a GET request.

> A malicious page that knows the url used in the email could open the url from the email in a popup

Sorry but this whole scenario is just ridiculous. If somebody can access your email it is already game over. It doesn't matter what web technology you are using at the that point, user interaction or not.

If a "malicious page" knows the URL it doesn't matter at all because it means it is capable of arbitrary code execution and that point it could just exfiltrate the URL to somebody or a Chromium instance to perform the user interaction. Actually if the page can open a popup I think it could also execute JavaScript within the context of the page and perform the user interaction right there.

So there's two scenarios:

Scenario a) No authentication-y bits in the url. User goes to the url, site checks if the user is already logged in via a cookie. If so, does the POST request.

Typically in this case the urls are easily guessable, so that's an easy CSRF. In principle they could be made per-user (some sort of HMAC on a user+timestamp). In practise, I think its fairly common for websites not to do that in this sort of situation.

scenario b) The url contains some sort of nonce, or signed assertion that automatically logs the user in. I think this is fairly common in email urls, because web developers want people to be able to take an action just from clicking the link in the email, even if the device they read email on is not the same as the device they normally use to interact with the application. I also think this is the scenario that applies to this discussion, since it was started around talking about the issues caused by google auto following links, and in scenario A, google auto-following links would not be an issue.

Of course, in principle, its possible that the authentication bits in the url, just authenticate that action, and don't generally log the user in. In practise I think its really common to just generally log the user in, since most sites want people to stay on the site once the user does anything, and not just immediately exit after the email action is completed.

These urls are typically not guessable. A small percentage of users do tend to smatter these across the internet (e.g. https://urlscan.io/), but ignoring that, this is a login-CSRF. That is, an attacker can generate their own such url, and force their victim to log into the attacker's account.

The impact of a login-csrf tend to be very application specific. Sometimes its kind of minor, but I have definitely seen cases in major websites where a login-csrf can lead to a full account take-over of the victim's account.

> Actually if the page can open a popup I think it could also execute JavaScript within the context of the page and perform the user interaction right there.

This is only true if the pop-up has the same origin as the site that opened it. Otherwise there is just a very limited API (Basically, postMessage(). Also both sides can change the current url of the other side, which is a bit nuts). Also there is now a new http header, Cross-Origin-Opener-Policy that affects this.

What you are describing isn't even CSRF

Yes it is - and it's worthwhile to read bawolffs well written explanation of how exactly it could be exploited. Downplaying security vulnerabilities of this sort is precisely how database leaks happen.

I don't downplay security issues. I just make sure they are actually understood first and that isn't what is going on here at all. What he is describing is not related to CSRF

Make the link password protected, and they don’t get the password in the same e-mail message.

Kinda hard to pre-scan a URL if you can’t provide the password for it.

Yeah lots of email clients do this. Next to caching also lots of scanners for malicious content.

We solved it by having a screen with a confirmation button , then later we added javascript to show a loader page over the button and click the button automatically.

That is a security risk that Google is causing here. While I agree that URLs shouldn't necessarily be used to store secrets, the usual password reset mail is nothing else and the mechanism has merit.

https://www.w3.org/TR/capability-urls/

It is also a good way to communicate between two parties that don't want to have user account in any service, we constantly request input from B2B customers by providing forms with a capability URLs. An no, we don't want to use an identity provider. Maybe good ones like auth0. Amazon Cognito is pretty decent in my opinion, but Amazon is also big tech. Industrial espionage is something real for that matter.

We have mail providers that respect privacy, just saying... I don't understand the love for Gmail at all, especially when you use a mail client, which I would heavily recommend to everyone.

Ironically a lot of security scanner also do follow links. Understandable, but I just hope they don't plaster the logs too much...

Another idea is to have 3 links, where only one is visible:

  https://example/com/token?forBots
  https://example.com/token
  https://example.com/token?forBots

Hopefully any automated systems will open the first or last link first, so that you can save the request info and filter based on that. In case requests come out of order, you can always add a small delay to the "human" link before responding.

I haven't yet gotten to implementing any of the authentication on my current project, so I might be missing something really basic.

The next best thing is to set a cookie when requesting the magic link, but the downside (or upside?) is that it will be valid only for the browser it was requested with.

No link in an email should perform an action on its own. Every link should lead to a confirmation button, at minimum. Too many services automatically open all the links in emails.

Tons of services send a verification link after registration, and when you click the link you are taken to a page that says "You're verified."

But in those cases there may be an automatic POST after you travel to the link, so it wouldn't be triggered by gmail looking up the url.

This may be for the purpose of ensuring the email address itself is deliverable. You don't want someone to sign up with random garbage, then try sending notifications, newsletters, etc. to it- I believe doing so can affect domain reputation.

For this use-case, it seems like even an automated link click would be a good signal of a deliverable email address.

Not just deliverable, but also that it's correct. There's a lot of people who think that my {firstname}{lastname}@gmail.com email address is their own. If they try to register it somewhere, a verification email stops them from completing the registration.

You can also check for various headers to determine (with quite good accuracy) if a link was clicked by a human or fetched programatically. Here's a list I've accumulated over the years for virtually the same feature as yours:

- `sec-fetch-dest` header is present (HUMAN)

- `accept` header is present (HUMAN)

- `from` header is bingbot(at)microsoft.com (AUTOMATED)

- `user-agent` header includes BingPreview (AUTOMATED)

HTH

Also most humans do use browsers, so if you don't have any following requests for resources like scripts, images or just the favicon, you probably got visited by a bot.

Not all humans use browsers that issue requests for additional resources.

I have my browser configured to retrieve the page only and no additional requests for CSS, images, or javascript.

Yay for building systems that rely on extremely non-idempotent behavior on GET.

We have the issue with 0bin.net burn after reading, so we give a grace period to the link after creation. It works decently, but I'm thinking of just displaying a page with a decrypt button on those, so that you need a post request to actually read the content and trigger the delete.

Consider a different approach to mitigate automated URL fetching interference (this can apply to both email ownership verifications and password resets).

Make the emailed verification/reset link (GET request) idempotent (1 and >1 request has the same effect).

Have the link just present an interface for the user to take the next step. In the next step make a POST request that actually commences your verification/reset process.

In all likelihood you'll want expiry logic (let's say it's 30 minutes) - if you store the token with a created_at timestamp on the server you can have your verification/reset process check that now < (created_at + 30 minutes)

If expired, provide a UI for the user to request a fresh verification/reset email.

Yes and outlook 365 too. We had to add an extra step to our activation process to handle this (a prompt to click a link to proceed). I would not be surprised if they start making their link-follower start clicking around inside opened pages too :-/

Here's a quick PoC:

  <link rel="prefetch" href="/actual_validate/email/1d00a5c2648c211befd33f5a8a7cbfab?prefetch=1">

  <script>
   location.href = "/actual_validate/email/1d00a5c2648c211befd33f5a8a7cbfab?js=1";
  </script>
  <noscript>
   <a href="/actual_validate/email/1d00a5c2648c211befd33f5a8a7cbfab?js=0" rel="nofollow" class="btn btn-primary" role="button">Click to confirm your account</a>
  </noscript>

Why the prefetch though? Reason being bots don't open them? If so, this is a really good idea!

It’s likely faster for the noscript case.

Yeah, I don't know about that. Wouldn't it be that the query strings differentiate the two links?

I assume so, because of an old trick where query strings are used for ad-hoc cache control as in /style.css?1629472765

You’re right, I wasn’t paying close enough attention. That’s what I get for reading HN mostly on my phone!

Query string is to differentiate the links (to understand which case is getting triggered)

OK, but then the prefetch link as it stands is useless, no?

A "GET" request is not supposed to alter state.

On the other hand. It validates the email address more quickly, so you could even refresh/poll when it's verified automatically

If there is a cookie session, 99% of GETs do alter the state.

I'm sure Google uses a specific user agent to make a request, so you can filter that out.

A better solution is to assume that some middleman (email server or client) will always try to access links in the email. Instead send the user a code and have them manually enter it on the linked page.

Or link them to a page with a POST form that actually performs the action. That way you only add a single click to the flow, and no remotely sane software will automatically perform POST requests to arbitrary urls.

> no remotely sane software will automatically perform POST requests to arbitrary urls

I'm not a web developer. Out of curiosity, why is that?

Websites are generally designed such that GET requests are side-effect free, and POST requests do have side effects. So for example searching on google is a GET request. It doesn't do anything other than serve the requested page. Logging in on the other hand has the side effect of setting cookies and probably writing some stuff to the database, so that's a POST request.

These assumptions are so baked into web software that while assuming a GET request won't do anything zany or overly stateful is probably fine, assuming the same for a POST request should probably be considered negligent.

For precisely the reason being discussed here: GET requests can be performed automatically for many reasons. For example, if you've ever pasted a URL into a Slack channel (or similar) and seen the link converted into a thumbnail of the page a few moments later, you've seen a piece of software issue a GET request on your behalf. Now imagine that wasn't a link to a page but a link to an something that modified your account - resetting your password, for example.

POST requests typically perform modifications on the server based on user action, like POST'ing this comment. GET requests should be idempotent.

They did not in my case. Here is the UA string. It looks like a normal client a user might have:

74.51.221.37 - - [19/Aug/2021:22:05:16 +0000] "GET /validate/email/1d00a5c2648c211befd33f5a8a7cbfab HTTP/1.1" 404 0 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36"

$ dig -x 74.51.221.37 +short

cache.google.com.

This is like the reason I quit using Skype 10 years ago. My colleagues and I noticed the same thing: send a link in a chat, and within seconds to minutes a request (or more) for that URL from a Microsoft server would be logged. Done and bye.

Wouldn't be surprised if pretty much every communication service is doing this.

I sent a link to a large file over Viber and immediately some ip connected and started downloading. Stopped at 350mb of around 3.5gb. I get that they want to show thumbnails or whatnot, but they just don't discriminate between content types.

That is a great way to do a cross-link DDoS of unsolicited link opening services - send 1000 messages with google links on skype and send 1000 gmail messages with microsoft links there, all gigabytes in size...

Does Skype generate preview images? Most chat clients do at this point. They all have to access the link to get the relevant metadata to do that.

Interesting "feature": the user perceives a (real) benefit: preview. But, there is also an unspoken benefit for Microsoft, who can feed their marketing analytics, or AI training data, or ... with what it learns via the chats and links.

I'd be fine if the fine print in the EULA provides a guarantee that the feature scans content solely for generating previews, and that M$ keeps no copy of it, etc....But, I'm sure I'd go blind looking for such text in the EULA.

This is interesting because it's something that would likely only happen in production or a staging server.

If you're building your web app in development, chances are your links will have localhost as their hostname which wouldn't trigger a visit from Google. You may also end up having an in memory fake email server to not even send the email in dev too (lots of web frameworks have solutions for this).

Checking the user-agent might work but I'm not a fan of this method because now it sets you up with having to keep a list of all known agents for every email client / service that might pre-visit URLs.

I’ve also seen anti-virus do this, though I don’t remember which brand.

You need to authenticate the user before the activation.

This. You could rely on a cookie during the get request as well, that you set on the users browser during registration. Or re-auth after click.

The problem with that is people like myself who tend to register on a laptop but then click the email verification link on their mobile phone.

(Because waiting for Gmail to load on a laptop is painful, whereas on my phone is shows up as a push notification within seconds)

What about magic links? :)

Does gmail respect robots.txt?

Not for this use case or crawling. They won't even hit robots. I've caught google, discord, valve, slack doing this on my hobby sites over the years, likely checking to see if the target URL contains obvious malware. In my case the solutions were simple. Add simple auth and/or block IP ranges associated with their AS number. Obviously this isn't the solution for companies, though you could have a unique domain specifically for email URL's and decide what limitations to put in place. Blocking them can flag your site as "malicious" and I am perfectly ok with google saying my domains are malicious.

Isn't there a way to verify that the click is coming from GMail? Maybe via User Agent or its IP.

The user agent I saw looks like a normal client that a person might use:

74.51.221.37 - - [19/Aug/2021:22:05:16 +0000] "GET /validate/email/1d00a5c2648c211befd33f5a8a7cbfab HTTP/1.1" 404 0 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36"

I suppose you could somehow block cache.google.com but I suspect Microsoft and others do similar things.

We've seen something similar with one of the email campaign services. The Unsubscribe links were "clicked" within a minute of emails being sent. Other services show you a message and process the unsubscribe on a POST request.

I've had a lot of grief from a few users' Exchange doing it (likely as part of some anti phishing plugin of sorts), to the point we changed validation links from one time to sort lived.

Don't know why you are downvoted. Many corporations and institutions employ sandboxes to check mails and the links contained in them. This is a standard security practice by now.

So a link that is only valid once would be affected. Restricting the validity by time is a good way to solve this while still maintaining decent security.

I presume robots.txt is still ignored in this case?

Make the link expire after a certain period of time and not after first click.

Block first attempt to access.

Haha that's one way to frustrate your users.

I had the same issue with Microsoft's email service and Facebook messages. How I dealt with it was to not email private links... I use Element these days or email links to encrypted files in some circumstances. I wish websites would stop using email and phone verifications...