Re:Ask HN: How do you feel about web scraping?
by tokenizerrr on Wednesday June 07, 2017 @07:28AM
(#14505319)
Were you seriously expecting bots to read your T&C? Or anyone, for that matter? Did you mention that it was okay for Google to scrape your site?
[ Parent
| on HN ]
Re:Ask HN: How do you feel about web scraping?
by throwaway6845 on Wednesday June 07, 2017 @07:32AM
(#14505344)
We're not talking generic "bots".
We're talking a custom scraper written for this site and this site only.
Yes, I am expecting the people who spend hours inspecting the source of my site, and then writing a custom scraper for it, to spend 30 seconds reading the T&Cs first.
Re:Ask HN: How do you feel about web scraping?
by tokenizerrr on Wednesday June 07, 2017 @09:57AM
(#14506457)
Not sure why you'd expect that. If my webbrowser can download your source code, my software will as well.
If you want people to read it put your content behind a sign up with a checkbox.
Re:Ask HN: How do you feel about web scraping?
by throwaway6845 on Wednesday June 07, 2017 @11:11AM
(#14507072)
It is _already_ behind a sign-up with a checkbox. They scraped their way past that too.
Re:Ask HN: How do you feel about web scraping?
by samstave on Wednesday June 07, 2017 @12:14PM
(#14507573)
How? (Seriously, how does one do this?)
Re:Ask HN: How do you feel about web scraping?
by kossae on Wednesday June 07, 2017 @01:54PM
(#14508446)
Simply log in first, then perform the scrape programmatically.
Seen here: https://kazuar.github.io/scraping-tutorial/
Re:Ask HN: How do you feel about web scraping?
by samstave on Wednesday June 07, 2017 @03:14PM
(#14509080)
oh... I thought they were able to circumvent logging in and could scrape directly... hat makes much more sense now, thank you...
Re:Ask HN: How do you feel about web scraping?
by tokenizerrr on Wednesday June 07, 2017 @11:27AM
(#14507197)
Ah, that changes things somewhat.