News for hackers. Stuff that matters.
Re:Parsing HTML Using Regular Expressions by monk_e_boy on Saturday July 01, 2017 @07:00PM (#14679572)
beautiful soup is a better tool.

Regex is fine for grabbing something in a page that you have looked at yourself.

Parsing millions of pages you don't have this option, you need something robust, a tool that is flexible, that doesn't barf out too many errors, that is quick.

Re:Parsing HTML Using Regular Expressions by mackerson on Saturday July 01, 2017 @07:37PM (#14679750)
BS4, requests and dataset are my go-to tools for simple scraping. Back when Bethesda pulled the paid mod nonsense on the Steam store I wrote a simple scraper to track and digest the changes people were making to their Skyrim reviews so I could tease out who was most offended, core fans with a lot of hours logged on the game or people who were just piling on the outrage bandwagon. I should dig back into that data some time.

Anyway, got the whole thing done in about 200 lines. Not bad to scrape 50,000 pages. Fun times.