Re:“Magic links” can end up in Bing search results, renderin...
by rsbadger on Monday June 27, 2022 @07:26AM
(#31893059)
Essentially yes - but "previewing" link and sending that link to Bingbot to crawl and index is another matter.
Imagine you send someone a "private" link to a file...Bing sees that and indexes it for the world to see. Not cool.
Re:“Magic links” can end up in Bing search results, renderin...
by swiftcoder on Tuesday June 28, 2022 @02:05AM
(#31904604)
Is this actually the Bingbot crawling for the index, or do they just use the same bingbot code to generate link previews?
If I have a well-tested crawler sitting here, I don't see why I'd write a brand new one just to fetch previews in outlook...
Re:“Magic links” can end up in Bing search results, renderin...
by Gigachad on Monday June 27, 2022 @07:45AM
(#31893193)
“Private links” should be covered by robots.txt. The only case I see this happening is for those “anyone with link” shares and those are easy to cover.
Re:“Magic links” can end up in Bing search results, renderin...
by rsbadger on Monday June 27, 2022 @07:55AM
(#31893278)
AFAIK, most of those "private" links are actually just "unlisted" but they're still public. I'm sure Bing is indexing those too...
Re:“Magic links” can end up in Bing search results, renderin...
by Gigachad on Monday June 27, 2022 @06:53PM
(#31901983)
Anything private should ideally be put behind authentication. If that isn't possible, than robots.txt. Search engines are _meant_ to index everything that is publicly accessible and not blacklisted on robots.txt.
Many email clients generate link previews so that they can display a thumbnail of the webpage. Would seem to be necessary to filter out those referrers from the validate link