Re:EU to unveil landmark law curbing power of tech giants
by absolutelyrad on Tuesday December 15, 2020 @11:08AM
(#25431526)
> Common Crawl
Not even close. I've used Common Crawl, it has nowhere near the breadth that Google has in their index.
Problem no. 1 is that website owners block non Google crawlers because of limited bandwidth. So you can't reliably even get started on creating your own search engine. Once we have the data available, we can figure out the other bits and what they require. But to get to that door, you need access to the crawl.
You don't need to have everything in the Google index to beat Google.
You need to be better than Google in the search niche that you'll serve.
And for that, at the bare minimum, you need access to all the crawl data that Google has to get started, then throw away the bits that you don't need.
Not even close. I've used Common Crawl, it has nowhere near the breadth that Google has in their index.
Problem no. 1 is that website owners block non Google crawlers because of limited bandwidth. So you can't reliably even get started on creating your own search engine. Once we have the data available, we can figure out the other bits and what they require. But to get to that door, you need access to the crawl.
You don't need to have everything in the Google index to beat Google.
You need to be better than Google in the search niche that you'll serve.
And for that, at the bare minimum, you need access to all the crawl data that Google has to get started, then throw away the bits that you don't need.