News for hackers. Stuff that matters.
Re:Only Google is really allowed to crawl the web by benatkin on Friday March 26, 2021 @12:08PM (#26593756)
That's incorrect. Before the search oligopolies formed, new search engines could start up. There was excite, hotbot, altavista, and more. Now they don't have access. Search these comments for census.gov.
Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @12:11PM (#26593802)
There are companies that do pretty well in this space, like ahrefs, for example. They do resort to trickery, like proxy clients that look like home computers or cell phones. But, if a small entity like ahrefs can do it, anyone can do it.

In a nutshell, though, I don't see equal access for all crawlers changing anything. Maybe that's the first barrier they hit, but it isn't the biggest or hardest one by far. Bing has good crawler access, but shit market share.