Re:Ask HN: "Creative" ways to handle abusive http traffic?
by bmpafa on Friday May 25, 2018 @10:31PM
(#17159834)
As someone often engaged in nonmalicious (but unwelcomed) scraping, this is one of the anti scraping measures I fear most
[ Parent
| on HN ]
Re:Ask HN: "Creative" ways to handle abusive http traffic?
by probably_wrong on Saturday May 26, 2018 @09:02AM
(#17161783)
Google did this to me. I wrote a script to find the cheapest flight in matrix.itasoftware.com (now Google Flights), and ran it once per hour. Pretty soon they started giving me only the most expensive flights, as in, 7K+ for a 1K flight.
Re:Ask HN: "Creative" ways to handle abusive http traffic?
by heyoni on Saturday May 26, 2018 @10:53AM
(#17162345)
That sounds like someone personally turned this on for you. Did you try to mask yourself and repeat?
Re:Ask HN: "Creative" ways to handle abusive http traffic?
by probably_wrong on Monday May 28, 2018 @05:54AM
(#17171567)
As funny as that would be, I doubt it was done personally against me - who would even take the time to block one person from getting one single cheap flight slightly cheaper?
But I didn't try anything fancy. In retrospective, I probably should have turned on my VPN and see what happened.
Re:Ask HN: "Creative" ways to handle abusive http traffic?
by Triesault on Saturday May 26, 2018 @12:26AM
(#17160214)
Out of curiosity, what kind of "nonmalicious but unwelcomed" scraping?
Re:Ask HN: "Creative" ways to handle abusive http traffic?
by geezerjay on Saturday May 26, 2018 @01:32AM
(#17160402)
Not GP but scraping price data off some stores (particularly travel sites) tends to be unwelcomed by operators.
Re:Ask HN: "Creative" ways to handle abusive http traffic?
by TeMPOraL on Saturday May 26, 2018 @06:14AM
(#17161247)
Myself, I did scrapping of a classified ads site once, to have notifications when a particular product I was looking for showed up for sale.
Re:Ask HN: "Creative" ways to handle abusive http traffic?
by scrollaway on Saturday May 26, 2018 @01:05AM
(#17160316)
Not GP but scraping data for personal projects off companies overprotective of their data is probably the most common example.
Re:Ask HN: "Creative" ways to handle abusive http traffic?
by Figs on Saturday May 26, 2018 @07:20AM
(#17161430)
Trying to get your entire reddit history, for example, is an obnoxiously difficult thing to do. It's impossible to get more than 1000 comments with the officially supported mechanism, leading people to do all kinds of strange workarounds to try to find older comments.
As a 10 year redditor, that really frustrates me, since I can't go back and see my early posts any more.
Re:Ask HN: "Creative" ways to handle abusive http traffic?
by toomuchtodo on Saturday May 26, 2018 @09:46AM
(#17161977)
Curious if Reddit will have to provide an easy mechanism to retrieve all of your comments now as part of GDPR.
Re:Ask HN: "Creative" ways to handle abusive http traffic?
by maxyme on Saturday May 26, 2018 @10:32AM
(#17162235)
Check the announcement post. In a comment one of the admins said they will eventually but for now it's email only for residents of the EU.
https://www.reddit.com/r/announcements/comments/8m2yr4/were_...
Re:Ask HN: "Creative" ways to handle abusive http traffic?
by duxup on Sunday May 27, 2018 @01:09AM
(#17165513)
If everyone in the EU could email them to speed this up that would be great.
Re:Ask HN: "Creative" ways to handle abusive http traffic?
by mathgeek on Saturday May 26, 2018 @08:33AM
(#17161696)
> As a 10 year redditor, that really frustrates me, since I can't go back and see my early posts any more.
I'd much rather they let you get your own history out and then make it impossible to get said history for anyone else (unless that person chooses to allow it).
Re:Ask HN: "Creative" ways to handle abusive http traffic?
by bmpafa on Saturday May 26, 2018 @03:53PM
(#17163941)
(GP here) re: the second part, I guess I consider most scraping 'unwelcome'
To the first part: I would characterize certain scraping, usually behind an authentication wall, as malicious--though admittedly that's not the right word. An example would be scraping Facebook profiles to build a marketing list.
So, by 'non malicious,' i mostly mean 'publicly available data'
Re:Ask HN: "Creative" ways to handle abusive http traffic?
by lostcolony on Saturday May 26, 2018 @06:34AM
(#17161299)
I've scraped content from sites with (and without) logins, for offline perusal when I was on a cruise.