> Microsoft does this because they're security scanning / checking all links in every Outlook email for known phishing and malware attacks

The problem with that is that the logic is broken. Microsoft cannot possibly know all phishing sites, especially for smaller things. By obfuscating the link the user can no longer verify it by themselves without clicking, but Microsoft will say it's safe. So the user is left with a false sense of security and are worse off.

It only works for huge sites ( e.g. mytwitter.lol phishing for twitter and similar), but drastically lowers the chance of less high profile phishing being caught.

The problem with that is that the logic is broken. If 99.99% of phishing can be prevented this way, what problem do you have with it? Would you really catch that 0.01% that an automated system wouldn't?

You mean you don't verify calls to action via other information channels? Fairly regularly I get phishing emails that correctly spoof the crypto headers of major sites (e.g., because of a misconfigured mail service). If an email asks me to do something, it either doesn't get done or I cover my ass in as many ways as possible, no exceptions.

That isn't by itself an argument against a good automated system -- I definitely like not having to sift through most of that garbage, but catching the 0.01% should be a routine practice, not something that seems like an insurmountable burden.

Maybe you should implement a "feature" that serves a simple static HTML page <p>This webpage is safe.</p> to "bingbot" and serve the real page to everyone else.

I believe the recommended practice is to hover over the URL before clicking the link.

If you do so, in Outlook, there will be a pop that shows "Original URL: XXX". This allows users to make a determination for themselves whether the link is safe or not.

We got some security courses about that too. Unfortunately, outlook replaces all of them with some safelink url rewriting, so the only way left to find out if a link is scammy is clicking it.

It is in fact possible to extract a destination URL from a Safelink one without clicking it. For the full link this can be tedious, but identifying the domain can still be done quickly.

For normal URLs, I agree. But in this case you have adversarial urls. Suppose the scammer puts some http and www.google.com in the url parameters, after some randomly generated 8 characters dot someobscuretld site.

I don't trust myself enough to be 100% sure I can decode an URLencoded misleading mess perfectly all the time.

They already hid urls in the username of the url, like www.google.com.unholymessherethatscrollsoutoftheurlbar @ malignantdomainnotgoogle.blah

Scammy Microsoft.

Microsoft offers this as a security product. It's impossible to know all links but known ones can be blocked to limit future issues. Other enterprise email security products scan the links and follow all the redirects as well. After delivery a incredibly small amount of time and every link is "clicked" in an email with those products.

Cool, so we should just stop building and running almost anything in existence because it's not all-encompassing? That sounds like a suboptimal path forward.

Scanning something for malware and publishing it in search results seem like 2 completely different things to me...?

But there is nothing to indicate either in the post or in the referenced SO thread that the URLs are published to the search results. They are visited by bingbot, that much seems confirmed, but there’s no example where one of these results shows up in the public search results.

They were indexed in Bing results, I’ve shared the URL of that in this thread

Holy shit thats bad. Do unlisted youtube and gdrive share links get indexed through this?

You’d assume those have proper robots.txt configuration?

I have a disallow all robots.txt for a production system. Have had from the beginning.

Bing indexes it. This is my first major security incident and I have no idea how to fix this without making everything totally shitty for the users.

Some services ignore global disallow, but will respect rules explicitly targeted at them.

I've put in a hard block for all crawlers on all pages. Works for my scenario I think. Hopefully they don't lie in their user agent. Then it's going to be really bad.

Yes, but there is no indication they are publishing it in the search results.

The original post is just complaining that the malware scanning is visiting the links.

They come to the following conclusion

>This effectively makes all one-time use links like login/pass-reset/etc useless.

Which we all know is not true because sites like onetimesecret.com allow for entering a separate password to prevent this sort of thing when it does happen.

It would be an interesting discussion to talk about what Microsoft's whitelisting process looks like, but the original article doesn't seem to understand what is going on well enough to drive the conversation in that direction.

They are publishing them - it has bitten us (e.g. expired one click links for customers ending up on Bing from their emails)

I think this is where we use meta tags.

All pages with one click links should have no index follow or no index no follow. Your seo consultant (if you have one) should have advised you on this.

I am not saying this excuses the privacy violation but just suggesting there are things we can do...

Bing appears to be ignoring those headers for links crawled from emails

Worse would be links that are private to the people who posses the url. Like a private video on YouTube or a private document in google docs. The security depends on the URL being secret. This would silently publish secret information.

If those pages have no proper meta tags or robots.txt, there’s absolutely nothing wrong with this. Security by obscurity was never a good approach; from Proxies to security scanners, there has always been software that crawls unassuming URLs and published the results somewhere, if only a report to the admin.

robots.txt disallow is ignored for my production site at least. This is super bad.

Same for us - we have robots.txt disallow etc. and the relevant headers for personal customer links and Bing is ignoring and publishing all the same

If you can say for certain that the links being published are coming from the malware scanning, and not being taken from users' browser sessions that are using Microsoft Edge you should elaborate on this.

I would be pretty mortified if browsers were using user browser sessions to scan content and pass it to bingbot…? What about if you’re browsing something local? Or your bank account?

I would be too.

The point I was making is that someone should research this instead of relying on wild speculation as the basis for the conversation.

That would be even worse.

Nobody is saying it isn't.

It's about trying to get to the core of the issue, not just the random speculation going on in the article and in this comment thread.

It is common for corporate email security appliances as well. URLs should not be used for authentication neither should email. I really want to pick brains of people that work on these types of systems to see why they don't think so.

Many people (most?) prefer to signup to services by email address. To do so, those email addresses must be verified. How would you verify it without sending them an email link?

You can verify validity of an email like that, no issue there. Just don't use that as a factor authentication. Control over an email account should not trump passwords (what you know) or proper 2fa (what you have, typically, email can be 2fa like sms and like sms it is not a good choice). If a person proves they control an email account then you ask them for additional info like secret questions or other information configured during registration.

I should not be able to take over your life because I compromised your phone which has sms, TOTP app and email.

a confirmation code?

Also, mail might not live on the same computer.

It doesn't matter if it's on the same computer. Sometimes all you need to do is click the link, not do anything on the page.

options include:

* use an interstitial page so that the actual activation is a POST request;

* send a confirmation code instead of a link

Is it ok to do a password reset through email? Because once you can do that you basically have email based authentication.

The password only makes this autentication less secure and it's not needed.

It is not. You can initiate password reser via email but additional recovery controls like security questions should still be required. In an ideal world you have 2fa as well, if you reset that via email as well the it isn't actually 2fa, it is email based 1fa with extra steps. If your 2fa has a separate mechanism for recovery as well,that would be ideal. If it was my webapp, I would use hashes of 3 answers to user chosen questions, hashes in the browser/client. It could be an object pairing as well, every user gets a list of 30 objects or so and they pick 3 pairs as a recovery combination.

Hashing an answer to many questions isn't prevent someone from guessing the input until the hashes match. So why bother hashing?

Recovery codes exist and are created at the time before recovery is necessary. But most people are going to lose their codes.

Who is going to remember what 3 things they picked out of 30 years ago?

If this were security scanning, why does it identify itself as BingBot? Doesn't that just allow cloaking and offer an easy workaround for any adversary with a modicum of intelligence?

I just love it when they "scan" password reset links.

The HTTP GET method is idempotent by specification. Visiting a webpage should not trigger password resets or any other actions by itself. If that's a problem then it's the site's fault for being defective.

You’re right that it was a bit of an oversight on my behalf, as the links were only generated after a verified human user action (signup) I had assumed the 1 time links to their email would be safe. But regardless of the link action, it shouldn’t be passing that data to Bingbot to crawl and (possibly) index in search engine results. Private email data should not be shared with search engine crawlers IMO.

So how do you implement a "one click unsubscribe" link in an email? They're on GET requests. You could use JavaScript on the resulting page to then trigger the unsubscribe but bots are now running JavaScript as well.

You show a webpage with an "Unsubscribe" button in it. The button triggers a POST request.

There's also RFC 8058: https://datatracker.ietf.org/doc/html/rfc8058

That should send one to a page with a confirmation button...

scanning with bing useragent? That's not a good idea.

Do they guarantee anywhere they’re not collecting this data to build profiles or do other analysis?

Wouldn't it be trivial to keep the list of malicious pages locally and not send any data?

You mean, push bing's entire list of malicious websites to every client? I doubt they want to or can do that.

And also use the local client to scan unknown links? They probably dont want outsiders to have access to this code.

Plus, how do you keep it from going stale?

If I were designing a system like this, I would not trust clients to perform legitimate analysis nor report legitimate results.

… but it doesn’t matter if the client is compromised, because all it would hurt is the user, right? If the client was compromised, it could just not send anything to your servers, or ignore the results, or …

What? But you're the one writing the client.

Doesn't matter. Never trust the client - it's outside of your control, it can be patched, it can be hacked, it can be spoofed, etc.

Little understanding: Undying trust of the client

Dunning-Kruger level of understanding: Never trust the client for anything ever, it's unreliable, everything must be off client.

Never mind the client is literally the interface into your system, so it being compromised is already game over for an application where the user is most vulnerable party you wanted to protect...

Deep understanding: Trusting the client requires a well thought out security model.

If the client is hacked in this case, they already have full control over what the user sees, they can cut out your remote check.

Maybe a good balance would be to hash the root of the URLs and compare those, or use fuzzy hashing on page contents, just so that the backend isn't getting a bunch of private urls that might accidentally get logged somewhere.

Trades detecting stuff hidden behind redirects for less liability on your backend, something to possibly consider depending on functional requirements.

It sounds like you’re advocating for no client at all

Just as a trivial example, how confident would you be in this auth scheme?

1. User opens Outlook and types in their email and password.

2. The app requests the user's password hash from the server and checks it.

3. Outlook tells the server auth was successful and gets a session token.

In this example you're right. For something like scanning a site for malicious content, on-device is not a bad approach. It decreases the amount of data sent to the server.

The client has a much bigger issue to worry about if the client-side malware scanning has been compromised. Malware could modify the UI/network calls such that your server-side scanning displays a positive result anyway.

You have to trust the client to display information to the user at some point. Link malware scanning that job can safely be delegated to the client. Authentication cannot.

My first guess is that giving phishers/scammers the list of all malicious domains/pages might allow them to circumvent it.

Does Gmail do this?

They definitely do link re-writing. As to what use they make of the original href attributes, I don't know.