News for hackers. Stuff that matters.
Re:Ask HN: How do you feel about web scraping? by clamprecht on Tuesday June 06, 2017 @07:21PM (#14502348)
Put the Terms and Conditions (the part relevant to scraping) in the /robots.txt as well.
Re:Ask HN: How do you feel about web scraping? by throwaway6845 on Wednesday June 07, 2017 @07:05AM (#14505158)
Yes. Did that after this episode.
Re:Ask HN: How do you feel about web scraping? by tokenizerrr on Wednesday June 07, 2017 @07:28AM (#14505319)
Were you seriously expecting bots to read your T&C? Or anyone, for that matter? Did you mention that it was okay for Google to scrape your site?
Re:Ask HN: How do you feel about web scraping? by throwaway6845 on Wednesday June 07, 2017 @07:32AM (#14505344)
We're not talking generic "bots".

We're talking a custom scraper written for this site and this site only.

Yes, I am expecting the people who spend hours inspecting the source of my site, and then writing a custom scraper for it, to spend 30 seconds reading the T&Cs first.

Re:Ask HN: How do you feel about web scraping? by tokenizerrr on Wednesday June 07, 2017 @09:57AM (#14506457)
Not sure why you'd expect that. If my webbrowser can download your source code, my software will as well.

If you want people to read it put your content behind a sign up with a checkbox.

Re:Ask HN: How do you feel about web scraping? by throwaway6845 on Wednesday June 07, 2017 @11:11AM (#14507072)
It is _already_ behind a sign-up with a checkbox. They scraped their way past that too.
Re:Ask HN: How do you feel about web scraping? by samstave on Wednesday June 07, 2017 @12:14PM (#14507573)
How? (Seriously, how does one do this?)
Re:Ask HN: How do you feel about web scraping? by kossae on Wednesday June 07, 2017 @01:54PM (#14508446)
Simply log in first, then perform the scrape programmatically.

Seen here: https://kazuar.github.io/scraping-tutorial/

Re:Ask HN: How do you feel about web scraping? by samstave on Wednesday June 07, 2017 @03:14PM (#14509080)
oh... I thought they were able to circumvent logging in and could scrape directly... hat makes much more sense now, thank you...
Re:Ask HN: How do you feel about web scraping? by tokenizerrr on Wednesday June 07, 2017 @11:27AM (#14507197)
Ah, that changes things somewhat.