Re:Brave Search replaces Google as default search engine in ...
by addingnumbers on Wednesday October 20, 2021 @12:13PM
(#28932469)
> if you don't crawl as frequently as google you are reliant on your crawl frequency being quicker than the 'content stealing' speed.
No problem there. If a page hasn't been crawled, it won't be a search result.
Re:Brave Search replaces Google as default search engine in ...
by Closi on Wednesday October 20, 2021 @05:54PM
(#28937274)
I mean you need to crawl often enough that you can determine which site stole content.
If a full crawl of the web takes 3 months and A steals content from B after 2 weeks, it may not be possible to easily infer that A has stolen from B rather than visa-versa.
If I crawl every week and see that A posted it on the first crawl, and B suddenly also had that content on the second crawl, I can infer that B stole from A.
If you recently started crawling you won't be able to detect which content was 'first', and if you don't crawl as frequently as google you are reliant on your crawl frequency being quicker than the 'content stealing' speed.