Re:Who exactly is crawling my site?
by toomuchtodo on Friday October 18, 2013 @03:11PM
(#6573380)
Could they not run webhooks where my site can call to them when to do a full or partial crawl? PubSub for web crawling, if you will.
[ Parent
| on HN ]
Sitemaps (http://www.sitemaps.org/) help a bit in that regard, as crawlers can check the sitemap and only crawl updated content.
Re:Who exactly is crawling my site?
by unhappyhippie on Friday October 18, 2013 @05:21PM
(#6574088)
Thanks.
PUsH: http://en.wikipedia.org/wiki/PubSubHubbub
Google is the only crawler I see using the protocol on my site. It does make Google updates occur within minutes, so that alone is reason to implement push.
Re:Who exactly is crawling my site?
by leephillips on Saturday October 19, 2013 @08:58AM
(#6576234)
I thought this was only being used for RSS - interesting!