News for hackers. Stuff that matters.
Re:The Web is missing an essential part of infrastructure: a... by Aeolun on Sunday April 21, 2019 @06:24PM (#19714962)
Google doesn’t crawl all of the internet very often either. Only sites that have proven to change a lot. So you could presumably supplement commoncrawl with your own more regular crawls.
Re:The Web is missing an essential part of infrastructure: a... by bogomipz on Sunday April 21, 2019 @09:46PM (#19715771)
I'm curious how they track and rank a site's "change velocity" without crawling all of the internet all of the time. It almost seems like a catch 22 no? Might you have any insight into how this works? Any suggested reading or links?
Re:The Web is missing an essential part of infrastructure: a... by jdsully on Sunday April 21, 2019 @10:59PM (#19716098)
A site you just learned about probably isn't very important, so you can just watch its month to month change. You don't care if they change that much because they aren't important yet.

A site that gets lots of links quickly (and is therefore important) will likely garner them from sites you are already frequently visiting.