News for hackers. Stuff that matters.
Re:Ask HN: Why do people use Puppeteer for webscraping inste... by billconan on Tuesday January 19, 2021 @03:52PM (#25838729)
my main concern is pricing. many websites use anti-scraping technologies. scraping the html doesn't work anymore. need to load everything and execute js. for example, I have seen some can detect headless / puppeteer mode too. I ended up creating my own scraping infra using vanilla chrome...

current saas platforms charge by request count. If I need to load everything, the cost will be too high.

Re:Ask HN: Why do people use Puppeteer for webscraping inste... by proszkinasenne2 on Tuesday January 19, 2021 @04:00PM (#25838828)
I thought about it too but when you consider cost of running headless Puppeteer (lets say on AWS) and the cost of a good proxy that is charged per GB its often as expensive (if not more) as some of these SaaS-es. This is the case especially for websites with some heavyweight JS/CSS/img assets.