News for hackers. Stuff that matters.
Re:EU to unveil landmark law curbing power of tech giants by ricardo81 on Tuesday December 15, 2020 @12:02PM (#25432169)
One possible issue is that the users of the data may place different prominence on a particular URL, and may decide that it warrants more frequent crawling.

For site owners, it may prove to be lower maintenance to simply whitelist such a bot for crawling, though on the downside they may want to pick and choose which search engines can index it, and since the crawl (assumedly) would use one UA it would complicate how site owners can tell respective search engines what they're allowed to do with the crawl data.

There are no comments on this comment yet.