News for hackers. Stuff that matters.
Re:How to scrape anything on the web and not get caught by salawat on Sunday April 22, 2018 @08:29AM (#16895880)
I'm kind of interested in an answer to this as well.

I know the typical "travel site" or "comparison shop" use case.

There is also the "darn it, I want this" use case.

However, automated, periodic web scraping that mutates (I.e. ticket or reservation grabbing bots) has always felt a bit squicky to me in the same way DNS squatting does.

Re:How to scrape anything on the web and not get caught by megous on Sunday April 22, 2018 @01:31PM (#16897282)
Website you like that invites discussion has a very shitty comment system, that doesn't notify you of the responses to your comments.

Or perhaps you want to monitor mentions of your name, and join the discussion.

Or you want to not lose your past thoughts, because discussions were deep and some may be interesting to re-read in the future.

Or a website is known to allow users delete their comments, or website itself bans users and hides all their content from others for flimsy reasons like "ISIS" in the title, no matter whether it's pro/against/neutral/irrelevant.

Or you want to organize the information differently than the site does.

Or the website is bloated and slow as hell, and you want to use it over gprs, so you create a lightweight/fast/better organized/filtered mirror.

Or the website doesn't have search or it is crap/slooow, and is not indexed in google, like some IRC logs out there.

Or you want to have content available offline.

...