A list of domains emitting [Terry Pratchett] clacks headers

Re:A list of domains emitting [Terry Pratchett] clacks headers by jonathanmh on Friday April 22, 2016 @08:04AM (#11548488)

Hi Jaruzel, the crawler is live :D you can see in the bottom which page it is crawling right now.

It's basically scraping all links on every page it hits and tests the headers if they are containing the clacks value.

I added the form so people can submit pages to speed up the development of the list, even though I believe eventually the crawler would get to their pages :D

[ Parent | on HN ]

Re:A list of domains emitting [Terry Pratchett] clacks headers by Jaruzel on Friday April 22, 2016 @02:34PM (#11551467)

Cool! What software are you using for the crawler?

[ Parent | on HN ]

Re:A list of domains emitting [Terry Pratchett] clacks headers by jonathanmh on Saturday April 23, 2016 @03:46AM (#11554604)

I built a not very advanced one with node.js and mongodb :)

Mainly in use in the crawler: * request * cheerio

[ Parent | on HN ]