News for hackers. Stuff that matters.

A comment on An open source API for web scraping.

Re:An open source API for web scraping by Jake232 on Wednesday June 24, 2015 @03:55PM (#9774073)
Cool idea. This could easily be extended to support something like a proxy pool; that way you can rate limit / rotate proxies for X domain globally at this server level. That way it's across all your projects, rather than having to do it on a per project basis.

Adding xPath support as well as CSS selectors would be a good addition.

[ Parent | on HN ]
Re:An open source API for web scraping by owainlewis on Wednesday June 24, 2015 @03:59PM (#9774100)
Will definitely do something with caching and rate limiting when I get some time. These queries are quite expensive so definitely needs a bit of work in those areas.
[ Parent | on HN ]