News for hackers. Stuff that matters.
Re:Congrats! Web scraping is legal! (US precedent) by powrtoch on Wednesday January 29, 2020 @09:36AM (#22180898)
"HiQ only takes information from public LinkedIn profiles. By definition, any member of the public has the right to access this information. Most importantly, the appeals court also upheld a lower court ruling that prohibits LinkedIn from interfering with hiQ’s web scraping of its site."

Surely I'm not reading this correctly. This would seem to suggest that websites are not legally allowed to prevent bots from crawling their sites. Lots of sites have ToS preventing such things, are those legally void now? Are captchas on public pages illegal, even if you request the page 8000 times in a second?

"In this case, hiQ argued that LinkedIn’s technical measures to block web scraping interfere with hiQ’s contracts with its own customers who rely on this data. In legal jargon, this is called” malicious interference with a contract”, which is prohibited by American law."

This is almost weirder. If LinkedIn wanted to force users to sign in to view profile info, would they be not allowed to do that because some company had signed a contract that implicitly assumed access to that data? If someone writes a web scraper for my site, and I unknowingly change my site in a way that breaks that scraper, can a court force me to revert the change?

Seems to imply that every business is somehow beholden to every contract signed by anyone.

Re:Congrats! Web scraping is legal! (US precedent) by nathanlied on Wednesday January 29, 2020 @09:55AM (#22181156)
> Lots of sites have ToS preventing such things, are those legally void now? Are captchas on public pages illegal, even if you request the page 8000 times in a second?

ToS are subservient to the law; you can (probably) terminate a service account from a user that breaks your ToS, but if the user does not have a service account (as is the case for HiQ, it doesn't seem they were using accounts for it), then your ToS does not apply, since you've technically not entered a binding legal contract with them.

> This is almost weirder. If LinkedIn wanted to force users to sign in to view profile info, would they be not allowed to do that because some company had signed a contract that implicitly assumed access to that data? If someone writes a web scraper for my site, and I unknowingly change my site in a way that breaks that scraper, can a court force me to revert the change?

IANAL, but I believe that'd fall on intent, and intent is often difficult to prove at a personal level, but not necessarily at a company level. If your intent for putting up barriers that happen to impact scraping, whatever they may be, was indeed to knowingly prevent scraping from a particular company, then you may be liable under this decision. This is the only part of the decision I'm torn on, since it's a bit messy to really prove such things. I'd be much more comfortable with allowing companies to take whatever measures they feel necessary to prevent scraping, and also allowing scrapers to legally circumvent those measures without threat of prosecution, assuming they didn't actually hack into anything.

Re:Congrats! Web scraping is legal! (US precedent) by andrewmutz on Wednesday January 29, 2020 @10:52AM (#22181821)
> but if the user does not have a service account (as is the case for HiQ, it doesn't seem they were using accounts for it), then your ToS does not apply, since you've technically not entered a binding legal contract with them.

Are you sure about this? I am not a lawyer, but I believe that the Terms of Service applies to all users, not just those that explicitly set up a user account.

I have interpreted the LinkedIn ruling to mean that scraping public data is no longer criminal activity but it still leaves you open to civil lawsuits for violating the ToS of the website you are scraping.

Re:Congrats! Web scraping is legal! (US precedent) by rohansingh on Wednesday January 29, 2020 @10:54AM (#22181859)
> Are you sure about this? I am not a lawyer, but I believe that the Terms of Service applies to all users, not just those that explicitly set up a user account.

How would that even work? If I browse to any random public page of your website, it's served to me before you've even transmitted the terms of service. How could I be bound by those terms of service when I haven't even seen them?

Re:Congrats! Web scraping is legal! (US precedent) by andrewmutz on Wednesday January 29, 2020 @12:42PM (#22183434)
As an engineer, I agree with what you are saying, but I think normal people and the courts disagree.

I think these sorts of contracts are called Adhesion Contracts (https://www.investopedia.com/terms/a/adhesion-contract.asp) and we interact with them all the time. For example, if you valet your car, the valet will hand you a piece of paper with a number printed on it to retrieve your car. On that paper you will find an adhesion contract that is valid and real (although not as powerful as the types of contracts that you sign)

Re:Congrats! Web scraping is legal! (US precedent) by AstralStorm on Thursday January 30, 2020 @11:45AM (#22192777)
This does not work at least for software licensing based on precedents for shrink-wrap contracts, so again would not work for licensing use of data.

A paper served you by the valet is not an immediate contract as you can deny agreeing to it and service does not happen.

You cannot do that with a publicly visible website, unless you show ToS and require agreement before first use. If you allow a non-transferable license then said data cannot be used by a search engine. If it's transferable you just pushed the problem towards scraping a different bot. (Well, you could have a direct agreement with a few major search engines.)

Caveat emptor: not a lawyer.

Re:Congrats! Web scraping is legal! (US precedent) by michaelmior on Wednesday January 29, 2020 @11:37AM (#22182449)
IANAL, but it seems like ToS could still govern your use of the data which you viewed. Sure, it seems like you couldn't claim any violation based on visiting a random page. But if the ToS is clearly identified on the page and you do something with the data that violates them, perhaps the owner of the site has a case.
Re:Congrats! Web scraping is legal! (US precedent) by cabaalis on Wednesday January 29, 2020 @11:56AM (#22182753)
> perhaps the owner of the site has a case.

Except it sounds like the owner doesn't. If the information is on the page made public, the owner of the page can't place terms on what is done with the data downstream. They'd have to implement some real binding system such as authentication where CFAA would apply. (IANAL)

Re:Congrats! Web scraping is legal! (US precedent) by turc1656 on Wednesday January 29, 2020 @12:58PM (#22183664)
Correct, but all of that is void if the data presented is any sort of protected information (copyright, IP, etc.). You can't, for example, scrape Yahoo Finance for pricing and dividend history and republish on your own stock tools website. They have a license to redistribute that data and publish on their own website. Similar story for copyrighted text and things of that nature.
Re:Congrats! Web scraping is legal! (US precedent) by AstralStorm on Thursday January 30, 2020 @11:53AM (#22192861)
That would require at least showing that ToS on first use. A link on a page is insufficient.

And said ToS would have to force copyright reassignment rather than a general licence, making LinkedIn culpable for any unlawful content published by users of its site.

Re:Congrats! Web scraping is legal! (US precedent) by checkyoursudo on Wednesday January 29, 2020 @03:10PM (#22185255)
I am a lawyer, and there isn't really an easy answer to these questions.

TOS are a lot like EULAs. If they look like contracts of adhesion, then they're going to get more scrutiny and skepticism. The TOS that you claim applies even to every single random visitor to your site where they do not in fact affirmatively agree to the terms is potentially going to look more like a contract of adhesion. That's a lot harder to enforce.

If they are used more for CYA so that you can ban undesirable accounts from your website which people explicitly agreed to when they signed up for it, or so that you can just up and alter your entire business model without having to give all of your customers refunds, then they're easier to defend.

Just my general opinion, of course. Every jurisdiction is different.

Re:Congrats! Web scraping is legal! (US precedent) by cthalupa on Wednesday January 29, 2020 @12:22PM (#22183148)
Also not a lawyer, but you cannot force me to accept your terms of service. Contract law requires both parties agree to enter it.

When you create an account, etc., you are agreeing to those terms. If I browse a public webpage that just has a terms of service link on the bottom of it, I've not agreed to anything.

Re:Congrats! Web scraping is legal! (US precedent) by elliekelly on Wednesday January 29, 2020 @12:18PM (#22183090)
> Are you sure about this? I am not a lawyer, but I believe that the Terms of Service applies to all users, not just those that explicitly set up a user account.

Typically you'll see TOS say something along the lines of "by continuing to access this site you agree..." or "if you do not agree with these terms you may not access this site..."

Whether that's enough to create a binding contract depends on the jurisdiction and who you ask.

Re:Congrats! Web scraping is legal! (US precedent) by checkyoursudo on Wednesday January 29, 2020 @03:02PM (#22185148)
It can also depend on the terms themselves. I can put "by using this site you agree to bake me a chocolate cake" on my website all day, but that doesn't mean I will be able to force you to bake me a chocolate cake.
Re:Congrats! Web scraping is legal! (US precedent) by beerandt on Wednesday January 29, 2020 @11:53AM (#22182712)
Terms of Service is a form of contractual agreement, which requires there be an offer and subsequent agreement by the parties.

I don't think criminal law was ever part of this.

Re:Congrats! Web scraping is legal! (US precedent) by andrewmutz on Wednesday January 29, 2020 @12:38PM (#22183371)
From the article, the LinkedIn decision was that scraping data does not violate the Computer Fraud and Abuse Act. Violating that act was considered to be criminal activity. (https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act)
Re:Congrats! Web scraping is legal! (US precedent) by beerandt on Wednesday January 29, 2020 @01:56PM (#22184434)
But the claim of a violation was only a claim as part of a civil trial. The law has both civil and criminal elements to it, and this is about the tort part of the law.

LinkedIn made threats accusing hiq of criminal behavior, but that doesn't mean there's any criminal precedent being set here, as far as I can tell. And no one was criminally charged.

Separately, part of the ruling states that for the purposes of authorization, defying a cease and desist letter does not constitute illegal access, which might have some criminal implications. They imply some sort of technical authorization system must be bypassed, which didn't happen, since the data is "public."

(Which doesn't square well, imho, with existing meatspace law. If a public serving business banned someone from their store, the door being unlocked isn't an excuse to ignore that ban and trespass. But I digress.)

With the overlapping areas of law, it's admittedly beyond my understanding. But the law is generally viewed, like dmca, as being overreaching, if not at least partly unconstitutional.

Re:Congrats! Web scraping is legal! (US precedent) by markandrewj on Wednesday January 29, 2020 @01:07PM (#22183788)
The CFAA is overreaching, and used often as a catch all. 'Reply All' has a good episode which explores this. This is actually what was used against Aaron Swartz when he was charged for downloading academic journals from MIT, and why his charges were unjustly severe.

Reply All - #43 The Law That Sticks https://gimletmedia.com/shows/reply-all/rnhoxb

Re:Congrats! Web scraping is legal! (US precedent) by markandrewj on Wednesday January 29, 2020 @11:05AM (#22181987)
It doesn't completely answer your question, but what Nathan is pointing out is that private contracts cannot negate common law.
Re:Congrats! Web scraping is legal! (US precedent) by ChrisMarshallNY on Wednesday January 29, 2020 @10:59AM (#22181916)
There's a long, long history (probably hundreds –if not thousands– of years old) of selling aggregated or processed publicly-available information.

I'm not particularly thrilled with it, but enough people think of it as a valuable enough service to pay for; even if they know they could get it themselves, for free.

LinkedIn users (as opposed to the company) might actually like what HiQ is doing, as it may help their own prospects.

Re:Congrats! Web scraping is legal! (US precedent) by ivanhoe on Wednesday January 29, 2020 @11:17AM (#22182157)
> but enough people think of it as a valuable enough service to pay for; even if they know they could get it themselves, for free.

It's not free, it takes time to collect data. Buying it makes a lot of sense as long as you pay less than what's your own time worth to you...

Re:Congrats! Web scraping is legal! (US precedent) by Cerium on Wednesday January 29, 2020 @11:25AM (#22182265)
It is true in the current situation, though I would prefer that we ensure free data must be free. In that case buyers of data would be incentivized to pressure providers of free data to improve the data quality.
Re:Congrats! Web scraping is legal! (US precedent) by moron4hire on Wednesday January 29, 2020 @11:58AM (#22182787)
The data does remain free, as long as LinkedIn still provides it for free.

The data without the noise is what you're paying for. The service of winnowing out what you care about from what you don't care about.

Considering how big of an effort it is, and that the source from which it came is still available, why should the cleaned data be free? If I collect fallen trees from public land, chop it into usable firewood, should my bundles of firewood also be free? Or I collect solar power with my own solar cells, should I have to give you the electricity for free?

Re:Congrats! Web scraping is legal! (US precedent) by munk-a on Wednesday January 29, 2020 @12:56PM (#22183632)
I think this is especially relevant when it comes to things that fall under disclosure & transparency requirements - a lot of information that is legally required to be made available isn't legally required to be convenient. So, as a patient, you may have the absolute right[1] to a free copy of the charge master[2] of a hospital you're admitted to but it could be required that you pick it up in person or that it is only supplied in microfliche form... so a company that's aggregated this and is reselling it can deliver real value.

1. This specific example is BS but plausible - I just wanted something more specific than the vagaries around things like FOIAs or shareholder reports which both have specific facts that can be rendered useless unless you have the context.

2. Basically, list of how much procedures cost.

Re:Congrats! Web scraping is legal! (US precedent) by ChrisMarshallNY on Wednesday January 29, 2020 @12:47PM (#22183503)
Absolutely spot-on.

I'm thinking of processed GIS data. If you have ever tried using the various formats that are supplied by government sites, you know what a huge pain it is.

I'm happy to pay a reasonable price for an interpreted and bowdlerized version.

Re:Congrats! Web scraping is legal! (US precedent) by moron4hire on Wednesday January 29, 2020 @12:59PM (#22183682)
I actually have! I had to import a huge file of all of the culverts around storm drains in a state, and each culvert was multiple pieces of geometry, none of them grouped together in any logical way. It was just a huge list of rectangles that looked like culverts when viewed visually but no way to identify them as being one culvert without heuristics on how close each rectangle was to others. Massively long process that should not have been so.
Re:Congrats! Web scraping is legal! (US precedent) by reroute1 on Wednesday January 29, 2020 @11:58AM (#22182794)
What do you mean free data must be free?

The data is free, but the aggregated formatted data has been worked on and processed, are you saying the resulting aggregated data should also be free? That isn't going to happen, why would anyone do that work for free?

Or are you afraid linkedIn and others will make everything private? That's completely up to linkedIn or individual linkedIn users what they want to make private vs public. Maybe more data would be made private if they don't want it scraped. I don't think that's inherently a good or bad thing.

Re:Congrats! Web scraping is legal! (US precedent) by michaelbuckbee on Wednesday January 29, 2020 @10:28AM (#22181542)
I'm trying to puzzle out how this works in practice. So if LinkedIn has truly public data (no login required to view) then it can be scraped no problem.

But if it's only accessible with a login, then it falls under TOS and they can be blocked?

Re:Congrats! Web scraping is legal! (US precedent) by matttb on Wednesday January 29, 2020 @10:13AM (#22181353)
> Surely I'm not reading this correctly. This would seem to suggest that websites are not legally allowed to prevent bots from crawling their sites. Lots of sites have ToS preventing such things, are those legally void now? Are captchas on public pages illegal, even if you request the page 8000 times in a second?

This is just a preliminary injunction. This wasn't an actual ruling on the case. This just says that until there is a ruling they can't stop the scraping to make sure the company isn't put under while waiting for an actual ruling.

Re:Congrats! Web scraping is legal! (US precedent) by remote_phone on Wednesday January 29, 2020 @10:42AM (#22181728)
You don’t understand what a preliminary injunction is then.

It’s a very, very strong indication that they will win. Courts don’t issue preliminary injunctions unless it’s extremely likely the side who won the preliminary injunction will win.

Re:Congrats! Web scraping is legal! (US precedent) by pbhjpbhj on Wednesday January 29, 2020 @11:07AM (#22182012)
Huh, I thought in USA they also did them to avoid an injunction having the effect of making the judgement irrelevant. So, where the case is not clear cut the injunction could prevent one party acting to 'kill' the other (and so avoid judgement) in the meantime?

Could you cite something on this that indicates this (my understanding here) is wrong?

Re:Congrats! Web scraping is legal! (US precedent) by wrs on Wednesday January 29, 2020 @11:15AM (#22182129)
It only requires a “substantial” likelihood that side will win (not an “extreme” one), which basically means there’s a substantive dispute. The more difficult criterion is a substantial likelihood that irreparable harm will occur if the injunction isn’t granted (irreparable harm is supposed to be a pretty extreme thing — it means you can’t fix it with any amount of money).
Re:Congrats! Web scraping is legal! (US precedent) by papln on Wednesday January 29, 2020 @12:13PM (#22183010)
Please read the linked decision before putting words in judges' mouths:

https://parsers.me/appeal-from-the-united-states-district-co...

Re:Congrats! Web scraping is legal! (US precedent) by coding123 on Wednesday January 29, 2020 @11:28AM (#22182308)
The issuance of an injunction is in no way related to how the future court battle will result.
Re:Congrats! Web scraping is legal! (US precedent) by munk-a on Wednesday January 29, 2020 @12:49PM (#22183540)
> This is almost weirder. If LinkedIn wanted to force users to sign in to view profile info, would they be not allowed to do that because some company had signed a contract that implicitly assumed access to that data? If someone writes a web scraper for my site, and I unknowingly change my site in a way that breaks that scraper, can a court force me to revert the change?

LinkedIn has long wanted to have their cake and eat it too - they advertised that data as being publicly accessible and allow google to index specific user pages but then attempt to restrict other bots from crawling it.

If you have private data behind a login there isn't an issue here - if you have public data but want some people to login before viewing it (or not be able to view it) then that's where this ruling comes up. So, this mostly hits sneaky SEO folks and dark UX patterns that rely on tempting someone with accessible data and then pulling the rug out from them at the last minute.

If your website places data outside of authentication then everyone should be able to see that data... I'm curious to see the specifics around

> Surely I'm not reading this correctly. This would seem to suggest that websites are not legally allowed to prevent bots from crawling their sites. Lots of sites have ToS preventing such things, are those legally void now? Are captchas on public pages illegal, even if you request the page 8000 times in a second?

though - DoS attacks are clearly illegal, but with this precedent there's going to be a lot of back and forth to see where the line between DoS and scraping falls... and I think that makes this precedent a lot weaker than the headline would have you believe. A company can still threaten to drag you through a lot of litigation by accusing you of malicious page requests, it'll take a few cases to define where that line needs to fall.

Re:Congrats! Web scraping is legal! (US precedent) by hiccuphippo on Wednesday January 29, 2020 @01:23PM (#22184016)
This reminds me about Twitter, when I click to see a thread for a tweet it asks me to login, but if I open the link in a new tab it loads the thread just fine.
Re:Congrats! Web scraping is legal! (US precedent) by username90 on Wednesday January 29, 2020 @10:15AM (#22181376)
Linkedin want their data to be scraped by bots so they have to keep it public, otherwise you wouldn't find peoples profile from Google. They just don't want bots from from their competitors like hiQ to scrape it.
Re:Congrats! Web scraping is legal! (US precedent) by danenania on Wednesday January 29, 2020 @12:17PM (#22183064)
To me, this is crucial. If it's public and available for google, it's public and available for everyone. If you want content to be private, then make it private and accept that you won't get search engine traffic. Otherwise, don't be surprised when your publicly accessible content is accessed by gasp the public.
Re:Congrats! Web scraping is legal! (US precedent) by papln on Wednesday January 29, 2020 @12:12PM (#22182997)
> hiQ argued

That does not mean that hte court agreed.

The judges said that CFAA doesn't apply.

In other words, the judges said that LinkedIn couldn't use the US legal system to force HiQ to stop. Judges didn't say that LinkedIn was barred from using technical measures.

The court did allow a preliminary injunction against LinkedIn, due to the possibility of "monopolies" (to be determined in Court later), pending resolution of that latter question.

LinkedIn might still win their claim to their right to block scrapers via technical means.

Re:Congrats! Web scraping is legal! (US precedent) by JoeAltmaier on Wednesday January 29, 2020 @12:16PM (#22183056)
Something about malicious interfering with a contract? That might prevent technical measures?
Re:Congrats! Web scraping is legal! (US precedent) by bena on Wednesday January 29, 2020 @12:33PM (#22183298)
They want to imply that, but they are wrong.

LinkedIn can't prevent HiQ from attempting to scrape their site through force of law.

LinkedIn can rate limit requests, make their site hard to scrape, change their format, whatever. LinkedIn is in no way responsible for how HiQ fulfills its contract to its customers. HiQ is attempting to say that if I sign a contract to provide you with a Tesla, then it would be illegal for Tesla to stop me from just taking one from them to give to you. If that sounds stupid, that's because it is.

Re:Congrats! Web scraping is legal! (US precedent) by scotty79 on Wednesday January 29, 2020 @09:53AM (#22181130)
> Seems to imply that every business is somehow beholden to every contract signed by anyone.

Implied contract is that if you publish something, it's public and you have no right to dictate what software people use to consume it.

Re:Congrats! Web scraping is legal! (US precedent) by phkahler on Wednesday January 29, 2020 @11:32AM (#22182386)
You also have no obligation to them.
Re:Congrats! Web scraping is legal! (US precedent) by lopmotr on Wednesday January 29, 2020 @10:02AM (#22181230)
The court document says "... refrain from putting in place any legal or technical measures with the effect of blocking hiQ's access to public profiles." on page 11. I wonder if they mean targeted measures specifically blocking hiQ but allowing others such as Google.
Re:Congrats! Web scraping is legal! (US precedent) by JackRabbitSlim on Wednesday January 29, 2020 @11:37AM (#22182441)
https://www.eff.org/cases/hiq-v-linkedin

LinkedIn aint the victim here...

Re:Congrats! Web scraping is legal! (US precedent) by hombre_fatal on Wednesday January 29, 2020 @01:33PM (#22184163)
This is the part I disagree with:

> hiQ also asked the court to prohibit LinkedIn from blocking its access to public profiles while the court considered the merits of its request. hiQ won a preliminary injunction against LinkedIn in district court, and LinkedIn appealed.

Whether LinkedIn is the good guy or bad guy here doesn't matter when the decision creates precedence for the rest of us.

Surely a healthier precedent is that we can respond arbitrarily to requests and have no obligation to the requester. So what if I want to randomize the html structure on every request or block requests from Tor because 100% of them are abuse? Can someone take me to court on the grounds that either is effectively "blocking" their scraping syndicate? Why not?

I feel like once CFAA is off the table (which I do agree with), the cat and mouse game is a fair middle ground. Keep web scraping a sport!

Re:Congrats! Web scraping is legal! (US precedent) by joering2 on Wednesday January 29, 2020 @12:09PM (#22182957)
Here is my attempt to draw an analogy.

There is a large banner next to the highway that shows some weather information that if properly organized (lets say monthly almanac) you would find people to pay money for it. The banner owner does not make money this way - he ask you to go to his website and signup for an account. But you drive the highway (internet) every day, look at the banner, write down the weather updates, and then offer them on your website as a sale. The owner gets angry and sue you. The court decides you are free to drive by the highway and you free to put your eyeballs on their weather banner, especially given the banner is available to everyone (LinkedIn profiles are avail to view without needing an account) and you are free to use the information you obtained for free without interference with said banner in a form of a monthly almanac that you sell. At the end of the day, the banner owner does not own the weather information that someone else put in there (for example a weather meteorologist).

I think personally its a healthy decision. Otherwise it would be similar to prejudice of who should be allowed to enter and browse a street store that by law is available to everyone.

Re:Congrats! Web scraping is legal! (US precedent) by candiodari on Wednesday January 29, 2020 @09:51AM (#22181103)
https://en.wikipedia.org/wiki/Tortious_interference

This would mostly mean that you cannot start interfering with webscraping you previously allowed merely because you learned that they're making money with the scraped data.

Re:Congrats! Web scraping is legal! (US precedent) by thfuran on Wednesday January 29, 2020 @10:36AM (#22181641)
It seems absurd if the 'interference' only directly affects their own property. Like, if my neighbors start monetizing livestreaming my backyard, suddenly I can't put up a fence? Except worse because in actuality, this third-party contract is costing them money through server load and bandwidth.
Re:Congrats! Web scraping is legal! (US precedent) by lovehashbrowns on Wednesday January 29, 2020 @11:25AM (#22182268)
Your analogy doesn't hold. Your backyard is private property. The data that LinkedIn publishes is intended for the public. That's why Google can index the pages and give you results from LinkedIn.
Re:Congrats! Web scraping is legal! (US precedent) by Supermancho on Wednesday January 29, 2020 @12:01PM (#22182846)
> Your analogy doesn't hold.

It does, in the US. You're likely making an inconsistent comparison.

Property ownership has nothing to do with visual access. You cannot legally be barred from casually (involuntarily) perceiving something. It's reasonable to put up physical barriers to reduce what is casually perceived. It's a very good analogy.

Re:Congrats! Web scraping is legal! (US precedent) by munk-a on Wednesday January 29, 2020 @01:25PM (#22184042)
However it doesn't hold - as your neighbor I can't bar you from putting up a fence because it'll intrude on my view of your property... granted people try to do that _all the time_ but I think it's commonly understood that putting up a fence for privacy is allowed.

It's also not a great analogy for this case because another party is given continued easy access to view my backyard while the first party is denied - and the analogy breaks down here because, as a neighbor, I have no inherent right to view your private life at least as much as any of your other neighbors.

Re:Congrats! Web scraping is legal! (US precedent) by Supermancho on Thursday January 30, 2020 @12:50AM (#22189162)
> it's commonly understood that putting up a fence for privacy is allowed.

Try building that fence into the stratosphere. A regulatory body will prevent that.

> I have no inherent right to view your private life at least as much as any of your other neighbors.

That's a different analogy, not a violation of the first.

It's not necessary for every part of the analogy to hold, being an analogy.

Re:Congrats! Web scraping is legal! (US precedent) by OJFord on Wednesday January 29, 2020 @12:06PM (#22182917)
It's trivial to fix that - the exterior of GP's house then. That's available for public viewing; is intended for it, but is private property. If you monetise livestreaming it and describe it in your ToS, GP can't repaint the front door, or get new windows?

Or perhaps slightly less contrived:

If I publish a monthly lowlights reel of my favourite sports team as a podcast discussion on where they can improve in all their lost games, and then they suddenly go on a winning streak for >1month so my USP is gone and I have nothing to talk about..?

Re:Congrats! Web scraping is legal! (US precedent) by jjeaff on Wednesday January 29, 2020 @12:31PM (#22183273)
Those examples don't fit because they are contracts not made in good faith. They aren't things you can control.

In this case, it was rules that the public data is available. It was a good faith contract on the part of HiQ to assume they could collect public data from a public website.

It would not be a good faith contract to assume you could control the paint colors on a property you don't own.

It seems to me that the interference ruling was wholly independent on deciding that what hiq was doing is legal.

Re:Congrats! Web scraping is legal! (US precedent) by papln on Wednesday January 29, 2020 @12:15PM (#22183042)
Does that mean that ia grocery store offers free samples, I can go in every day and take all the samples, and the grocery is not allowed to selectively prevent me access?
Re:Congrats! Web scraping is legal! (US precedent) by munk-a on Wednesday January 29, 2020 @01:22PM (#22183994)
It means that if they're offering free samples and refuse to offer you the same service they're offering to other customers they might be in hot water - which is consistent with what a lot of folks consider ethical. Offering an item for free to some folks and not to others is a form of discrimination - it's usually not a particularly troubling form of discrimination but in this case Google is allowed to walk up and take all the samples and the grocery store manager just smiles and nods - but when you (hiQ in this example) try and get one you're hit with an injunction and barred from entry.
Re:Congrats! Web scraping is legal! (US precedent) by kazagistar on Wednesday January 29, 2020 @12:19PM (#22183095)
Can someone taking down a open source project, like the leftpad debacle, be sued for tort?
Re:Congrats! Web scraping is legal! (US precedent) by cthalupa on Wednesday January 29, 2020 @12:27PM (#22183215)
I mean, anyone can be sued for anything. I can file a lawsuit with basically zero legitimacy to it. It'll probably get thrown out, but you were still sued.

If the question is could someone win, potentially. The argument would basically have to be that the removal of that open source project is akin to other cases of negligent interference.

If this is a specific concern, consult a lawyer - 'cause I'm not one.

Re:Congrats! Web scraping is legal! (US precedent) by jjeaff on Wednesday January 29, 2020 @12:33PM (#22183303)
Doubtful. If linkedin had completely taken down their site, or put it all behind a login account, them this case would not have turned out the same.

Maybe if leftpad somehow tried to block only some users from using their publically available plugin.

Re:Congrats! Web scraping is legal! (US precedent) by KptMarchewa on Wednesday January 29, 2020 @11:18AM (#22182163)
User data is not theirs property.
Re:Congrats! Web scraping is legal! (US precedent) by frandroid on Wednesday January 29, 2020 @11:31AM (#22182354)
Your backyard can be a walled garden--this is about the public front of the property.
Re:Congrats! Web scraping is legal! (US precedent) by candiodari on Wednesday January 29, 2020 @12:05PM (#22182897)
Exactly your backyard is of course yours. But you are not at liberty to use it to damage others. There's lots of rules about this. For example, opening a brothel on your own land is definitely not legal without considering how it affects the neighborhood.
Re:Congrats! Web scraping is legal! (US precedent) by papln on Wednesday January 29, 2020 @12:16PM (#22183048)
When is it "damage", and when is it "declining to contribute" ?
Re:Congrats! Web scraping is legal! (US precedent) by tcd on Wednesday January 29, 2020 @10:44AM (#22181741)
If I decide to change the class names, or HTML structure of the page, is that no longer allowed?

How far does this go?

Re:Congrats! Web scraping is legal! (US precedent) by yorwba on Wednesday January 29, 2020 @11:11AM (#22182075)
Are you doing it just to spite scrapers, i.e. with "malicious intent"? If you have some other reason, you won't be guilty of intentional tortious interference.
Re:Congrats! Web scraping is legal! (US precedent) by byuu on Wednesday January 29, 2020 @12:16PM (#22183051)
> "Most importantly, the appeals court also upheld a lower court ruling that prohibits LinkedIn from interfering with hiQ’s web scraping of its site."

How would this affect Cloudflare's "checking your browser" anti-DDoS protection screen, meant to block bot requests from accessing sites?

Re:Congrats! Web scraping is legal! (US precedent) by dalore on Wednesday January 29, 2020 @11:41AM (#22182524)
So if I (like many others) have cloudflare web scraping protection turned on, that is now against American law?
Re:Congrats! Web scraping is legal! (US precedent) by papln on Wednesday January 29, 2020 @12:14PM (#22183017)
There's no reason to believe that.
Re:Congrats! Web scraping is legal! (US precedent) by etxm on Wednesday January 29, 2020 @11:35AM (#22182424)
It will be interesting to see how this will impact the bot detection market like perimeterX, etc.
Re:Congrats! Web scraping is legal! (US precedent) by protanopia on Wednesday January 29, 2020 @12:41PM (#22183413)
> If LinkedIn wanted to force users to sign in to view profile info

Do they not already do this? Every link I've ever seen for LinkedIn has redirected me to sign up page rather than showing me the content.

Re:Congrats! Web scraping is legal! (US precedent) by thrwn_frthr_awy on Wednesday January 29, 2020 @12:48PM (#22183518)
They want search engines to index their profiles and provide organic search results links to their site, but then those same sites will require you to sign in when clicking a link to another public profile. You can search for that 2nd profile in Google and then view it without signing in, but not by clicking internal links. I've experienced this with Quora, LinkedIn, Instagram, FB and others. They want to have their cake and eat it too.
Re:Congrats! Web scraping is legal! (US precedent) by alok-g on Wednesday January 29, 2020 @01:00PM (#22183687)
As a user of LinkedIn, I can pick which portions of my profile information I would like to be publicly available. This is not by default, so most people do not have it public. You can try seeing my profile without logging in. :-)
Re:Congrats! Web scraping is legal! (US precedent) by MarioMan on Wednesday January 29, 2020 @12:52PM (#22183576)
They block you out after the first few profiles you view. Try a private browser and you can still see them.
Re:Congrats! Web scraping is legal! (US precedent) by giarc on Wednesday January 29, 2020 @09:48AM (#22181051)
Your second point is interesting. I suspect the contract between hiQ and some company is that hiQ provides info on public profiles, and if LinkedIn removes all public profiles by requiring a login the contract would become moot. Just the same if I was to change my profile settings from public to private, hiQ wouldn't be in breach of their contract (nor would I).
Re:Congrats! Web scraping is legal! (US precedent) by onetimemanytime on Wednesday January 29, 2020 @09:57AM (#22181174)
Scrapping should either be legal or not. The fact that you have a contract to sell the content you assumed it's legal to scrape, should not matter. Too bad if you lose money
Re:Congrats! Web scraping is legal! (US precedent) by jacobwilliamroy on Wednesday January 29, 2020 @12:47PM (#22183514)
>Are those [ToS] legally void now?

They were pretty much legally void even before this precedent was established. They are only valid when they don't violate any existing U.S. law. Any authority assumed beyond that is completely false.

>can the court force me to revert that change?

No.

Re:Congrats! Web scraping is legal! (US precedent) by wtetzner on Wednesday January 29, 2020 @09:54AM (#22181144)
I wonder if it has anything to do with the fact that the data is actually owned by LinkedIn users, and they expressed that they want their data to be publicly available?
Re:Congrats! Web scraping is legal! (US precedent) by shadowgovt on Wednesday January 29, 2020 @10:37AM (#22181652)
Unlikely. The license to LinkedIn retains ownership, but the user's retention of information ownership doesn't compel LinkedIn to affirmatively do things with that data (i.e. LinkedIn isn't forced to vend the data to a given consumer if the user says so).

The license further goes on to clarify that LinkedIn will vend public data to search engines, but the definition of "search engine" is almost certainly assumed (by LinkedIn, at least) to be up to them.

Re:Congrats! Web scraping is legal! (US precedent) by sjg007 on Wednesday January 29, 2020 @09:54AM (#22181148)
It's because previously public information became private. They can't do that apparently.
Re:Congrats! Web scraping is legal! (US precedent) by vectorEQ on Wednesday January 29, 2020 @11:02AM (#22181946)
i'm wondering if that robots.txt might then get you sued due to blocking scrapers / bots?
Re:Congrats! Web scraping is legal! (US precedent) by herval on Wednesday January 29, 2020 @11:30AM (#22182334)
A robots.txt file doesn’t block scrapers, it’s the equivalent of a no trespassing sign. I don’t think putting up signs will suddenly become illegal?
Re:Congrats! Web scraping is legal! (US precedent) by setr on Wednesday January 29, 2020 @11:29AM (#22182324)
Robots.txt doesn't interfere with anything. It's a suggestion.