why aren't Larry and Sergey behind bars, then? Scraping publicly available information is far from illegal.

Also, Interestingly only Alibaba's bots are completely blocked from crawling: https://www.amazon.com/robots.txt

> Scraping publicly available information is far from illegal.

The scraping itself may not be (although I'm pretty sure here in Belgium there is a law against collecting other people's data), but what you do with it may not be legal.

You could make a case for making any kind of profit generated from scraping data illegal. Don't get me wrong, I love scraping things myself.

Also find it amazing there are companies out there like Crawlera that can do serious scraping work and openly flaunt deploying tech to get around whatever scraping blockers are out there.

>

LinkedIn had multiple layers of scraping detection systems deployed, and went to significant efforts to block their data from getting scraped[1].

Last year, they were ordered by a Federal court explicitly to allow scraping of content and remove systems that were designed to impede and block scraping efforts[2].

There's no clear law (in the US) directly aimed at scraping, and repurposing anti-hacking laws brought up the murky definition of what is unauthorized access. If a judge clarifies explicitly that scraping is not unauthorized access (which happened in this case, although needs to stand up to appeal[3]), then entities that are interested in preventing scraping have lost one of their core legal underpinnings. It demonstrates why companies like Crawlera have been able to flaunt the type of serious scraping work they do, because it's hard to bully people with a legal argument that has been debunked and affirmed as debunked on appeal. So it's better to avoid the risk of setting that precedence entirely until you can't avoid it.

[1] https://techcrunch.com/2016/08/15/linkedin-sues-scrapers/

[2] https://www.reuters.com/article/us-microsoft-linkedin-ruling...

[3] https://www.courthousenews.com/linkedin-takes-data-scraping-...

Check amazon api T&C, also try to do the same with Craigslist and see how long you they will let you do it. scraping data is always a shady business if you do it without a permission of content owner

It is anything but shady. They can send you a C&D or file a suit and seek injunction but there is no way they can get you in trouble with the law for scraping publicly available data.

I don't need to check the T&C to view the page.

If you’re scraping.. you’re not using the API.

Why would the owner of a product want to keep their product info a secret?

Ex. People take products data and copy to eBay then try to dropship getting products from your fba. People pay big money for nice photos of products and then somebody just comes and takes it as their own

> People pay big money for nice photos of products and then somebody just comes and takes it as their own

This raises an interesting question: if someone had a product on Amazon and had product photos they took, does Amazon still allow other sellers to use the same listing? In other words, does the seller agreement allow Amazon to reuse your (potentially expensive to produce) product photos on your competitor's product listing?

Yes they allow it. It often leads to "product listing hijacking": another seller sends a counterfeit/similar product to amazon using the UPC of your (private label) product. Product & review pages are then shared between all these different products. AFAIK the only way to combat this is purchasing the other product, document how if differs from your product, report it all to amazon and hope they act before too much damage is done to your product listing.