News for hackers. Stuff that matters.
Re:Who exactly is crawling my site? by toomuchtodo on Friday October 18, 2013 @03:11PM (#6573380)
Could they not run webhooks where my site can call to them when to do a full or partial crawl? PubSub for web crawling, if you will.
Re:Who exactly is crawling my site? by ceejayoz on Friday October 18, 2013 @03:17PM (#6573412)
Sitemaps (http://www.sitemaps.org/) help a bit in that regard, as crawlers can check the sitemap and only crawl updated content.
Re:Who exactly is crawling my site? by unhappyhippie on Friday October 18, 2013 @05:21PM (#6574088)
Thanks.
Re:Who exactly is crawling my site? by sergiosgc on Saturday October 19, 2013 @03:58AM (#6575726)
PUsH: http://en.wikipedia.org/wiki/PubSubHubbub

Google is the only crawler I see using the protocol on my site. It does make Google updates occur within minutes, so that alone is reason to implement push.

Re:Who exactly is crawling my site? by leephillips on Saturday October 19, 2013 @08:58AM (#6576234)
I thought this was only being used for RSS - interesting!