Ferret – Declarative web scraping

Re:Ferret – Declarative web scraping by holoduke on Tuesday October 02, 2018 @02:52PM (#18124549)

The best way to scrape already for many years is using a headless browser plugin. For example phantomjs with nodejs. That in combination with tor or a large proxy pool is unbeatable by all other alternatives.

[ Parent | on HN ]

Re:Ferret – Declarative web scraping by ziflex on Tuesday October 02, 2018 @02:54PM (#18124566)

This is how it works under the hood. But everything is wired for you ;)

[ Parent | on HN ]

Re:Ferret – Declarative web scraping by jotadambalakiri on Tuesday October 02, 2018 @03:35PM (#18124978)

I wouldn't scrape over Tor, you would be slowing down the network for people who actually need it. Maybe if you are running a node.

[ Parent | on HN ]