Re:LinkedIn: It’s illegal to scrape our website without perm...
by kstrauser on Monday July 31, 2017 @05:01PM
(#14895860)
Many years ago, I wrote a scraper for a certain .mil financial website. My company at the time - and its customers - had legitimate accounts on that website, but the site was horrible to use. Page loads took upwards of a minute, and there were no front-page notification systems to let you know if any of the information on the site had changed. Kind of a big deal when it was a window into the government's accounts payable system. For instance, if you didn't notice that an auditor had left a comment on an invoice, and you didn't respond to it within X hours, then the gov't could deny the invoice and you'd have to start over. The end result was our customers paying full-time employees to manually examine every single open invoice in the .mil system every single day, which just increased load on the website and made it even slower.
Enter my scraper. It copied data into a local PostgreSQL database that our customers could run reports against. A process that used to take a human 6 hours a day now took 30 error-free seconds. The scraper was even a benefit to the website as we ran it overnight during low-load conditions, and because my software was smarter than a web browser, we could retrieve the same information as a human with about 1/3 the number of web requests. Perfect, right!
Well, no. We got an angry call from the developers complaining that we were the ones making their site slow, even though 1) we measurably didn't, 2) we were lighter on resources than the humans we were replacing, and 3) we scrupulously obeyed their rate limiting requirements and erred on the side of running 10% slower than they had originally requested from their customers.
That particular problem went away when we pointed out that their shiny new website wasn't Section 508 accessibility compliant and could not be made to be without literally throwing out their entire web service and starting over, but our website was, and that if they continued to allow us to serve screen reader compatible pages to our disabled customers then there wouldn't be a need to have the .mil website shut down and a Congressional investigation launched. All parties involved decided this was a reasonable compromise.
Enter my scraper. It copied data into a local PostgreSQL database that our customers could run reports against. A process that used to take a human 6 hours a day now took 30 error-free seconds. The scraper was even a benefit to the website as we ran it overnight during low-load conditions, and because my software was smarter than a web browser, we could retrieve the same information as a human with about 1/3 the number of web requests. Perfect, right!
Well, no. We got an angry call from the developers complaining that we were the ones making their site slow, even though 1) we measurably didn't, 2) we were lighter on resources than the humans we were replacing, and 3) we scrupulously obeyed their rate limiting requirements and erred on the side of running 10% slower than they had originally requested from their customers.
That particular problem went away when we pointed out that their shiny new website wasn't Section 508 accessibility compliant and could not be made to be without literally throwing out their entire web service and starting over, but our website was, and that if they continued to allow us to serve screen reader compatible pages to our disabled customers then there wouldn't be a need to have the .mil website shut down and a Congressional investigation launched. All parties involved decided this was a reasonable compromise.