Re:Robots.txt is a suicide note
by derpy on Tuesday May 10, 2011 @03:12AM
(#2531685)
You do try to contact the "targets" sometimes though?
[ Parent
| on HN ]
Re:Robots.txt is a suicide note
by sbierwagen on Tuesday May 10, 2011 @03:16AM
(#2531690)
Sure. We ask them to give us database dumps, when they refuse, for some crazy reason, we just download their website.
I can't say I approve of the "if the crazy fools won't give us what we want, we'll just take it some other way" attitude.
As the "some crazy reason" might be that the database contains data that is not publicly addressable for a reason and sanitising the data-set could be non-zero effort operation, letting you crawl the public pages and do some work at your side would be the sensible route from most points of view. Heck, even if there is definitely not anything not already displayed to the public in the DB, letting you go ahead and get the information the hard way is likely to be preferable to lifting a finger to send you a DB dump.
Sort of like a cat's attitude to playing fetch: You want it? You go get it. I've got more important things to do.