News for hackers. Stuff that matters.
Re:Web Scraping with Python by lapser on Sunday May 15, 2022 @01:47PM (#31389375)
Not really. The API these days tends to be JSON so you can just figure out how to it works and what represents what.

For example I've been able to reimplement xmltv scrapers for several sources in less than a 100 lines with Scrapy. It's not hard, just requires a little discretion.

Re:Web Scraping with Python by chrsig on Sunday May 15, 2022 @03:25PM (#31390229)
The difficulty isn't in making a scraper for a single site, but rather in the general case.

That is, making a scraper that can be pointed at an arbitrary site not known at the time of development.

Re:Web Scraping with Python by vertere on Sunday May 15, 2022 @10:14PM (#31393138)
I assumed most people were talking about dealing with single sites. From your previous comment about API documentation and "hand evaluating" Javascript I gathered that you were too. How would those things help one solve the general case?