Re:Notes on Writing Web Scrapers
by gringo_flamingo on Monday December 06, 2021 @07:46AM
(#29459040)
Thank you for your reply, and your feedback, man! I will be sure to take this knowledge with me on my next web scraping journey! I appreciate your time XD
Thinking about separation of concerns helped me a lot in getting over the hump of perfectionism. Once I realized I was trying to make my software do too much, it was easier to see how it would be much less work to write as two separate programs bundled together. (Talking specifically about the extract/transform stages here.)
Upon reflection, this project has been just as much self-study of good software engineering practices as it has been learning how to scrape job sites XD