News for hackers. Stuff that matters.
Re:Ask HN: How do you feel about web scraping? by throwaway6845 on Tuesday June 06, 2017 @07:03PM (#14502239)
I built a simple CRUD app for a previous (small) employer. Nothing special technology-wise, but a good concept, sound business model, and backed up with a couple of full-time staff creating content for it. Line one of the T&Cs was "no scraping". Business model was based on sales to individual users but we were prepared to do analysis in aggregate if asked.

A scraper company, funded by magic money (Knight Foundation grants) and $1m of VC, convinced a (UK) Government department to pay them to scrape our site for some analysis the department wanted. They'd never contacted us, never asked for permission, never asked if we could supply the data. Our company was bumping along at this point and having to lay people off. Income from a nice lucrative Government contract would have kept a couple more people in work.

The scraper company's FAQ was, in my view, full-on unethical:

> "we check the robots.txt file. If the site permits robots in general to scrape their site (NOT just GoogleBot!), then we will do so. We will make no effort to look for other terms and conditions as well."

You will ostentatiously "make no effort to look" for T&Cs in case they prohibit the significant contract you're about to sign with the Government? Whoa.

So how I feel about web scraping is simple: "don't be evil". If you're diverting income or traffic from the original site, don't do it. If you're genuinely adding value, go for it, but be open, be prepared to work with the original site, and be prepared to accede to their wishes.

Re:Ask HN: How do you feel about web scraping? by clamprecht on Tuesday June 06, 2017 @07:21PM (#14502348)
Put the Terms and Conditions (the part relevant to scraping) in the /robots.txt as well.
Re:Ask HN: How do you feel about web scraping? by throwaway6845 on Wednesday June 07, 2017 @07:05AM (#14505158)
Yes. Did that after this episode.
Re:Ask HN: How do you feel about web scraping? by tokenizerrr on Wednesday June 07, 2017 @07:28AM (#14505319)
Were you seriously expecting bots to read your T&C? Or anyone, for that matter? Did you mention that it was okay for Google to scrape your site?
Re:Ask HN: How do you feel about web scraping? by throwaway6845 on Wednesday June 07, 2017 @07:32AM (#14505344)
We're not talking generic "bots".

We're talking a custom scraper written for this site and this site only.

Yes, I am expecting the people who spend hours inspecting the source of my site, and then writing a custom scraper for it, to spend 30 seconds reading the T&Cs first.

Re:Ask HN: How do you feel about web scraping? by tokenizerrr on Wednesday June 07, 2017 @09:57AM (#14506457)
Not sure why you'd expect that. If my webbrowser can download your source code, my software will as well.

If you want people to read it put your content behind a sign up with a checkbox.

Re:Ask HN: How do you feel about web scraping? by throwaway6845 on Wednesday June 07, 2017 @11:11AM (#14507072)
It is _already_ behind a sign-up with a checkbox. They scraped their way past that too.
Re:Ask HN: How do you feel about web scraping? by samstave on Wednesday June 07, 2017 @12:14PM (#14507573)
How? (Seriously, how does one do this?)
Re:Ask HN: How do you feel about web scraping? by kossae on Wednesday June 07, 2017 @01:54PM (#14508446)
Simply log in first, then perform the scrape programmatically.

Seen here: https://kazuar.github.io/scraping-tutorial/

Re:Ask HN: How do you feel about web scraping? by samstave on Wednesday June 07, 2017 @03:14PM (#14509080)
oh... I thought they were able to circumvent logging in and could scrape directly... hat makes much more sense now, thank you...
Re:Ask HN: How do you feel about web scraping? by tokenizerrr on Wednesday June 07, 2017 @11:27AM (#14507197)
Ah, that changes things somewhat.
Re:Ask HN: How do you feel about web scraping? by jlebrech on Wednesday June 07, 2017 @04:20AM (#14504531)
you could rate limit the site and when a limit is hit replace paragraphs with lorem ipsum.
Re:Ask HN: How do you feel about web scraping? by cheetos on Wednesday June 07, 2017 @03:37AM (#14504365)
Did your service offer a paid API? Scraping happens because of a lack of better options. Surely you can understand why the scrapers didn't want to contact you beforehand.
Re:Ask HN: How do you feel about web scraping? by pimmen89 on Wednesday June 07, 2017 @04:46AM (#14504630)
If you want my data on a paid API basis, then ask me about it. I need to know how big the demand is for third party users before I even prioritize building a paid API, having the god damn courtesy to ask for something would give me an idea.

If you're using my data to hijack my traffic, without asking, you could have all the right justifications in the world but you're still a prick. Who knows, maybe your orphanage building app will move me tears once I hear about it and I'll give you free access.

Re:Ask HN: How do you feel about web scraping? by throwaway6845 on Wednesday June 07, 2017 @07:06AM (#14505163)
In what world is "engage a third-party scraping company" a better option than "drop a quick email to the site operator"?
Re:Ask HN: How do you feel about web scraping? by binarymax on Wednesday June 07, 2017 @07:10AM (#14505184)
Because with scraping you are in a legal grey area. But if you contact the site directly and they say "no", then there is no excuse to scrape.
Re:Ask HN: How do you feel about web scraping? by throwaway6845 on Wednesday June 07, 2017 @07:12AM (#14505201)
Well, yeah, like if you ask someone to sell you their dog and they say "no". Doesn't justify stealing the dog.
Re:Ask HN: How do you feel about web scraping? by binarymax on Wednesday June 07, 2017 @07:19AM (#14505255)
Your analogy doesn't hold up. Your example is clearly theft, and is a criminal matter. Violation of a sites terms is a civil one, and again is a legal grey area. Scraping the site doesn't delete the content from a server...but there is only one copy of the dog.

EDIT - I can't reply to your comment below, but FWIW I agree that scraping sites in this manner is unethical. I am merely describing the logic that most scrapers go through for self justification and legal protection.

Re:Ask HN: How do you feel about web scraping? by throwaway6845 on Wednesday June 07, 2017 @07:30AM (#14505331)
The analogy was "doing something anyway because you might be told 'no'", not "my server behaves like a canine quadruped". Copyright infringement is also potentially criminal in the UK, it's not as simple as you suggest.

But whatever. It just saddens me that the internet is a constant "don't be a dick" battle with companies like the scraper guys.

(edit - understood :) )

Re:Ask HN: How do you feel about web scraping? by jbreckmckye on Wednesday June 07, 2017 @03:48AM (#14504407)
Yes, because they did not want to pay. This is not some complex issue.

Incidentally, I'll be shopping later. May I give you $5 for you to drive to London and take me there? I was going to steal your car, but then I figured I'd be a good citizen and demand you provide it to me at your own, prohibitive, loss.