Re:Reinstate hystry's Hacker News
by akkartik on Wednesday January 16, 2008 @06:54PM
(#99545)
Great idea. I always knew my crawler was a bit of a hack and would be eventually obsoleted.
Until we get there, though, may my crawler be let back in? Like I said in email I promise to only restart it after I embed controls over how many URLs I get per hour deeply inside it, and to log my crawling activity and keep an eye on it in future.
The reason I was crawling every few minutes was to get notified of updates to conversations as they happen, before the 'herd' moves on to new feeding grounds. But I can make it crawl even hourly and it'll still allow me to stay kinda abreast of the conversation here.
Until we get there, though, may my crawler be let back in? Like I said in email I promise to only restart it after I embed controls over how many URLs I get per hour deeply inside it, and to log my crawling activity and keep an eye on it in future.
The reason I was crawling every few minutes was to get notified of updates to conversations as they happen, before the 'herd' moves on to new feeding grounds. But I can make it crawl even hourly and it'll still allow me to stay kinda abreast of the conversation here.