News for hackers. Stuff that matters.
Re:Notes on Writing Web Scrapers by gringo_flamingo on Thursday December 02, 2021 @02:11PM (#29420963)
I can truly relate to this article, especially where you mentioned trying to extract only the specific contents of elements that you need; without bloating your software. To me, that seemed intuitive with the minimal experience I have in web scraping. However, I ended up fighting the frameworks. Me being stubborn, I did not try your approach and kept trying to be a perfectionist about it LMAO. Thank you for this read, glad I am not the only one who has been through this. Haha...
Re:Notes on Writing Web Scrapers by cushychicken on Thursday December 02, 2021 @02:26PM (#29421165)
Yeah it's an easy thing to get into a perfectionist streak over.

Thinking about separation of concerns helped me a lot in getting over the hump of perfectionism. Once I realized I was trying to make my software do too much, it was easier to see how it would be much less work to write as two separate programs bundled together. (Talking specifically about the extract/transform stages here.)

Upon reflection, this project has been just as much self-study of good software engineering practices as it has been learning how to scrape job sites XD

Re:Notes on Writing Web Scrapers by gringo_flamingo on Monday December 06, 2021 @07:46AM (#29459040)
Thank you for your reply, and your feedback, man! I will be sure to take this knowledge with me on my next web scraping journey! I appreciate your time XD