News for hackers. Stuff that matters.
Re:More notes on writing web scrapers by marginalia_nu on Friday February 25, 2022 @08:47PM (#30474574)
I'd note, regarding storing data, that HTML compresses extremely well. It will shrink to <10% of its size. If you want to save it, you would be a fool if you saved it uncompressed.
Re:More notes on writing web scrapers by cushychicken on Saturday February 26, 2022 @07:17AM (#30476978)
Guess I'm a bit of a fool then because I was definitely saving uncompressed haha.

Maybe I'll give it a try at some point, but right now, I don't have any need for it. It'd be work without a clear payoff. I'm starting a new job at the end of March, and I'd rather concentrate on some features that help me run the site between now and then.