News for hackers. Stuff that matters.
Re:More notes on writing web scrapers by marginalia_nu on Friday February 25, 2022 @08:47PM (#30474574)
I'd note, regarding storing data, that HTML compresses extremely well. It will shrink to <10% of its size. If you want to save it, you would be a fool if you saved it uncompressed.
Re:More notes on writing web scrapers by cushychicken on Saturday February 26, 2022 @07:17AM (#30476978)
Guess I'm a bit of a fool then because I was definitely saving uncompressed haha.

Maybe I'll give it a try at some point, but right now, I don't have any need for it. It'd be work without a clear payoff. I'm starting a new job at the end of March, and I'd rather concentrate on some features that help me run the site between now and then.

Re:More notes on writing web scrapers by cushychicken on Friday February 25, 2022 @12:24PM (#30469663)
Author here. Let's talk web scrapers.
Re:More notes on writing web scrapers by sockopen on Friday February 25, 2022 @12:51PM (#30469979)
Copying a request as cURL, PowerShell, fetch etc… right from inside the DevTools is a blessing.
Re:More notes on writing web scrapers by cushychicken on Friday February 25, 2022 @01:00PM (#30470103)
Isn't it though?

Does create a bit of work when you have to figure out which parts of cURL need to be ported to Python, and which can be safely omitted. Copying a cURL request adds in a lot of headers - many of which I still don't properly know the purpose of.

I'll get there eventually. : ) In the meantime - thank god for whoever wrote "Copy as cURL request"!

Re:More notes on writing web scrapers by dnnssl2 on Sunday February 27, 2022 @05:33AM (#30487008)
Is there such thing as an unscrapable site? I tried to open driver.uber.com with Pyppeteer and it fails. I’m guessing it’s due to redirects, so what have you seen solve this problem?
Re:More notes on writing web scrapers by cushychicken on Tuesday March 01, 2022 @03:23PM (#30519473)
Workday is my 800 pound scraping gorilla.

Never heard of Pyppeteer.