Re:Parsing HTML Using Regular Expressions
by mackerson on Saturday July 01, 2017 @07:37PM
(#14679750)
BS4, requests and dataset are my go-to tools for simple scraping. Back when Bethesda pulled the paid mod nonsense on the Steam store I wrote a simple scraper to track and digest the changes people were making to their Skyrim reviews so I could tease out who was most offended, core fans with a lot of hours logged on the game or people who were just piling on the outrage bandwagon. I should dig back into that data some time.
Anyway, got the whole thing done in about 200 lines. Not bad to scrape 50,000 pages. Fun times.
Regex is fine for grabbing something in a page that you have looked at yourself.
Parsing millions of pages you don't have this option, you need something robust, a tool that is flexible, that doesn't barf out too many errors, that is quick.