Re:Parsing HTML Using Regular Expressions
by mackerson on Saturday July 01, 2017 @07:37PM
(#14679750)
BS4, requests and dataset are my go-to tools for simple scraping. Back when Bethesda pulled the paid mod nonsense on the Steam store I wrote a simple scraper to track and digest the changes people were making to their Skyrim reviews so I could tease out who was most offended, core fans with a lot of hours logged on the game or people who were just piling on the outrage bandwagon. I should dig back into that data some time.
Anyway, got the whole thing done in about 200 lines. Not bad to scrape 50,000 pages. Fun times.
Anyway, got the whole thing done in about 200 lines. Not bad to scrape 50,000 pages. Fun times.