Re:Notes on Writing Web Scrapers
by yakshaving_jgt on Friday December 03, 2021 @09:43AM
(#29430531)
I think you have a case of premature optimisation there, as I wrote in a recent comment[0].
Re:Notes on Writing Web Scrapers
by cushychicken on Friday December 03, 2021 @10:25AM
(#29431050)
Not sure what's premature here. The optimization is to allow me, a human, to find a certain class of database records quickly. I also chose a method that I understand to be snappy on the SQL side as well.
What would you suggest as a non-optimized alternative? That might make your point about premature optimization clearer.
Re:Notes on Writing Web Scrapers
by yakshaving_jgt on Friday December 03, 2021 @11:59AM
(#29432216)
There is indeed a trade-off, and the direction I would have chosen is to use meaningful status names as opposed to magic numbers. My reasoning being that maintenance cost in terms of how self-explanatory the system is makes more sense to me economically than obscuring the meaning behind some of the code/data for a practically non-existent performance benefit.
After all, hardware is cheap, but developer time isn't.
For a more concrete example, I might have chosen the value `'pending'` (or similar) instead of `5`. Active listings might have status `'active'`. Expired ones might have status `'expired'`, etc.
I have a field, post_status, in my backend database, that I use to categorize posts. Each category is a numeric code so SQL can filter it relatively quickly. I have statuses for active posts, dead posts, ignored links, links needing review, etc.
It's a way for me to sort through my scraper's results quickly.