Re:Writing Less Code
by derefr on Wednesday August 17, 2016 @07:59PM
(#12309468)
I know it's a solved problem now, but:
> One that couldn't easily be changed, either, because everyone's clients relied on the exact structure of the markup for screen scraping.
This doesn't seem like that hard a problem. Presumably the robots all sent their own distinct UA strings (as scrapers only get written to imitate browser UA strings if you ban non-browser UA strings.) So why not just filter your logs to find all the known scrapers, serve anything with those exact UA strings "legacy" table-based responses, and serve everyone else the new layout?
> One that couldn't easily be changed, either, because everyone's clients relied on the exact structure of the markup for screen scraping.
This doesn't seem like that hard a problem. Presumably the robots all sent their own distinct UA strings (as scrapers only get written to imitate browser UA strings if you ban non-browser UA strings.) So why not just filter your logs to find all the known scrapers, serve anything with those exact UA strings "legacy" table-based responses, and serve everyone else the new layout?