Parsing Wikipedia Articles with Node.js and jQuery

Re:Parsing Wikipedia Articles with Node.js and jQuery by decad on Sunday August 26, 2012 @07:11PM (#4436608)

This is very interesting, although anyone aiming to crawl Wikipedia should make sure they read this section on the Database download page. http://en.wikipedia.org/wiki/Wikipedia:Database_download#Why...

Everything should be fine as long as you respect their 1 request per second rule and their robots.txt

[ Parent | on HN ]

Re:Parsing Wikipedia Articles with Node.js and jQuery by kefs on Sunday August 26, 2012 @09:09PM (#4436931)

A quick skim of the source shows that rate-limiting is not implemented, and the code is non-compliant with Wikipedia's crawling rules.

[ Parent | on HN ]

Re:Parsing Wikipedia Articles with Node.js and jQuery by BenjaminCoe on Sunday August 26, 2012 @10:15PM (#4437110)

Thanks for the heads up. I'll add rate-limiting directly into the API.

[ Parent | on HN ]