Cleaning up bad bots

Re:Cleaning up bad bots by eli on Monday September 23, 2019 @04:50PM (#21053474)

Not sure what "legitimate" means here? Legal? Running aggressive web crawlers is in many instances against the rules for consumer cloud servers. For example, AWS requires that you obey robots.txt if you run a crawler there. https://aws.amazon.com/premiumsupport/knowledge-center/repor...

In my experience a lot of bots seem to be running on hacked servers or through hacked/insecure proxies. I'd imagine tracking down the owner or someone upstream of those boxes could be effective in taking them offline.

[ Parent | on HN ]

Re:Cleaning up bad bots by ikeboy on Monday September 23, 2019 @04:52PM (#21053495)

What does that have to with my point? Bots used to purchase inventory (and that aren't otherwise commiting fraud by using stolen credit cards or something) are not malicious.

[ Parent | on HN ]

Re:Cleaning up bad bots by coldpie on Monday September 23, 2019 @05:57PM (#21054198)

> Bots used to purchase inventory are not malicious.

There's no way you're in this conversation without being aware that scalping is a controversial practice at best.

https://theconversation.com/the-economics-of-ticket-scalping...

https://en.wikipedia.org/wiki/Ticket_scalping

[ Parent | on HN ]

Re:Cleaning up bad bots by ikeboy on Monday September 23, 2019 @06:00PM (#21054223)

I'm well aware that many economically illiterate people like to scaremonger about scalping.

That doesn't make them right.

[ Parent | on HN ]

Re:Cleaning up bad bots by coldpie on Monday September 23, 2019 @06:04PM (#21054244)

Or perhaps they are interested in optimizing the process for something different than what you are optimizing for.

[ Parent | on HN ]

Re:Cleaning up bad bots by ikeboy on Monday September 23, 2019 @06:08PM (#21054270)

Maybe I'd believe that if they stopped saying things like scalping harms the producer, and acknowledged that they want a less efficient market.

[ Parent | on HN ]

Re:Cleaning up bad bots by JakeTheAndroid on Tuesday September 24, 2019 @08:39AM (#21059414)

if a single bot controller can buy up an entire stock of limited items legitimately, that is malicious as that company is not longer able to meet the needs of their consumers. That's bad for the company.

[ Parent | on HN ]

Re:Cleaning up bad bots by ikeboy on Tuesday September 24, 2019 @10:39AM (#21060658)

If it's profitable for anyone to resell, that implies the company priced below the market price and there would be a shortage without scalpers. So the company is unable to meet the needs of its customers in any event. Scalpers just make it somewhat more efficient.

[ Parent | on HN ]

Re:Cleaning up bad bots by judge2020 on Monday September 23, 2019 @06:06PM (#21054255)

It is illegal if the website’s TOS for making a purchase prohibits the use of automated software.

It doesn't matter if it's legal, it matters if the website owner doesn't want x doing y on their site. A bot consistently not abiding by owners' intent is inheritly malicious.

[ Parent | on HN ]

Re:Cleaning up bad bots by ikeboy on Monday September 23, 2019 @06:10PM (#21054283)

What law does using a bot to make a purchase against ToS violate?

The LinkedIn case shows that it's not illegal in some cases - would you classify the behavior of the defendant in that case as malicious?

[ Parent | on HN ]

Re:Cleaning up bad bots by eli on Monday September 23, 2019 @05:00PM (#21053591)

Are they following the sneaker website's robots.txt while doing that? If not, they are probably violating the AWS terms regardless of whether you believe that activity is "malicious."

[ Parent | on HN ]

Re:Cleaning up bad bots by ikeboy on Monday September 23, 2019 @05:06PM (#21053656)

if they're running on AWS, which most crawlers are not

When I've run scraping software in the past I used DigitalOcean, which doesn't contain a requirement to abide by robots.txt. As far as I can tell it's both legal and consistent with their ToS to run a program that makes purchases on a website.

[ Parent | on HN ]