Re:The Fight to Mine Your Data and Sell It to Your Boss
by skybrian on Wednesday November 15, 2017 @07:34PM
(#15709343)
Traditionally this is controlled by robots.txt, which does let you control which search engines can scrape your site. It's not enforceable, but generally considered fair play to respect it.
I think it's well within a website's rights to control who scrapes them. The problem with LinkedIn (and other social media sites) is that they don't let their users control this, in turn. It should be up to each user to decide whether they want to appear in search engines, and which ones.
It's pretty bizarre that this was never brought up in the article. The idea that users should control their own information was entirely ignored.
I think it's well within a website's rights to control who scrapes them. The problem with LinkedIn (and other social media sites) is that they don't let their users control this, in turn. It should be up to each user to decide whether they want to appear in search engines, and which ones.
It's pretty bizarre that this was never brought up in the article. The idea that users should control their own information was entirely ignored.