Re:“Magic links” can end up in Bing search results, renderin...
by scandinavian on Monday June 27, 2022 @08:24AM
(#31893552)
The URL in the search is this: https://shoprocket.io/email-confirmation/34b35b1...
I don't see that in the robots.txt https://shoprocket.io/robots.txt
User-agent: *
Disallow: /cdn-cgi/l/email-protection
Disallow: /login
Disallow: /register
Disallow: /404
Am I missing something?
Re:“Magic links” can end up in Bing search results, renderin...
by rsbadger on Monday June 27, 2022 @08:39AM
(#31893708)
You may be seeing a stale version, try this: https://shoprocket.io/robots.txt?bypass=1
(I made a lot of changes today when testing all, including "visit as Bingbot" from their webmaster tools with and without the URL blocked by robots.txt)
https://www.bing.com/search?q=https%3A%2F%2Fshoprocket.io%2F...