It gets into the incredibly murky water of how the web works. You're just issuing a request and getting things back. Sometimes in a web browser, sometimes not. But the content itself may still be copyright. You can't just take it, even though for now, the publisher/server is allowing you to view it for free.

But what if you only chose to view some of the content (e.g. block ads). What if you apply your own styles to change the way that information is displayed? You're just changing the way the browser represents that data. You're not redistributing it as your own at this point. What if you store that data, but don't republish it; just used it in some each algorithms?

There are a whole lot of interesting grey areas here, but many that already have precedents that side more with the copyright holders.

The interesting part here is that linkedin doesn't hold any copyright on much of the data. You cannot copyright someones name and title.

But you can copyright a database.

that one is iffy, http://www.nolo.com/legal-encyclopedia/types-databases-eligi...

I'd suspect LinkedIn could argue about the network they create is a creative work and would be covered, but the facts about each person might not be copyright-able.

whats inside the database is not owned by them.

Maybe the whole idea of copyright is flawed and harmful?

There's nothing wrong with copyrights; 14 year copyrights.

Why 14, though?

I'm well aware of the historical precedent, but that number was rather arbitrary even then - it was what a bunch of people agreed upon, based on their ideas and experience, and given the environment. It's doubly arbitrary today, considering how much the environment has changed. Is a 14-year copyright on software reasonable, for example, or too long.

Rather than making it a hard cut-off point, it would be interesting to come up with a scheme that attempts to capture the spirit of term limits.

Consider: why are copyright terms even a thing? Well, copyright is a monopoly on a thing that is not naturally restricted; it does not exist in the absence of society, and is therefore a privilege granted by that society. By itself, copyright is meant to encourage creativity in the interest of public good, and at the same time, to provide some means to derive profit from one's creative expression. So there are two conflicting interests at play here - the desire of the creator to be rewarded for the fruits of his labor, and the desire of the society to enjoy growing, constantly enriched culture. The copyright term, then, marks the point at which the latter trumps the former.

Instead, what we could do is capture the fact that the interests conflict. For as long as you hold copyright, you're effectively denying society the ability to freely enjoy the culture that you have enriched. Why, then, not tax the copyright accordingly? You could consider it a kind of intellectual property tax, but with a twist: the longer copyright is held, the more the interests of society are infringed, and the larger the compensatory payment required to maintain the copyright.

So we could start with a grace period of a couple of years that is completely free, then it starts growing steadily. For some really popular work that makes significant profits, the author could easily afford payments to maintain copyright for a decade or two (or however long - that is something that can be dialed arbitrarily). For things that are too obscure, payments would cease shortly, and they would fall to public domain. There wouldn't be such a thing as "abandonware" anymore.

What use to put the money to? Many possibilities there. Publicly sponsored arts and art education is an obvious choice. Another interesting example would be offering bulk sums of money to authors of culturally important works to surrender their copyrights sooner, so that the public can enjoy them.

I'd love to see more debate about just this.

> It gets into the incredibly murky water of how the web works.

There's no "murky water" in how the web works. It's very clear and precise, and anybody can learn how it works. It has to be precise and well defined, because computers can't operate any other way.

If Linkedin doesn't want "public" profile data to be accessible to everybody then they need to stop calling it public and put it behind some kind of access control.

But that's exactly what LI did, then came a judge and ordered them (temporarily) to remove said access controls.

Computers are also operating with absolute clarity and precision when somebody exploits a flaw in their software to execute arbitrary code. That's why the CFAA has its language around 'exceeding authorized access' (even though it turns out that definition is dangerously vague).

I think most of those "grey areas" already have plenty of precedent in the world of art & music. Have you heard of "fair use", and what it allows and does not?

There is no grey area. You can not copyright facts. If you download ("scrape") a webpage and then extract the facts, whatever you downloaded only exists in volatile memory. So there is no claim there. The only claim you can make is on the download itself, hence what LinkedIn chose.

I've seen plenty of LinkedIn profiles that could be classified as fiction.

But LinkedIn isn't the owner of that fiction.

It's possible that HiQ is downloading and processing things that aren't facts. Long passages of text in LinkedIn posts, recommendations, and comments aren't "facts".

Though that doesn't appear to be the path LinkedIn is using to fight it.

Copyright isn't thoughtcrime, if they aren't redistributing it to anyone with standing to sue (very likely not LinkedIn, no matter their ToS) they can process things all day.

I did not say or imply "thoughtcrime". Just noting it can be more complex. Sentiment analysis of copyrighted text passages might be claimed as being a derivitive work for example. Fair use does have limits.

The standing to sue is an issue for user generated content, yes.

>Sentiment analysis of copyrighted text passages might be claimed as being a derivitive work for example.

So if a company releases a product that predicts likelihood of an employee quitting, you think you're going to have standing to sue because an analysis of a copyrighted passage you wrote comprised 0.000001% of the source material the algorithm was trained on?

You left out my last sentence, on purpose I suppose.

The context was that scraping doesn't always get a free pass because "facts". This specific case may skirt it because of the user generated content. Doesn't mean it's not worth mentioning for the larger context that copyright isn't black and white.

This case is much more significant than that. The ruling will determine whether or not breaching the terms and agreements of a website constitutes a crime under the Computer Fraud and Abuse Act (CFAA). LinkedIn is arguing that since scraping is not defined by the company itself to be a permitted use, a federal felony is being committed by anyone using a script to access its website. The language used in the CFAA is "unauthorized access" and this case will set a legal precedent for the meaning of that term.

So some company can sue me if I use `wget somecompany.com` if LinkedIn win?

We can hope that this case will set a legal precedent, but it may, like the Craigslist case, end in a settlement (or some other disposition) before that point is reached.

If a website puts something on the public internet, it should not even be aware if it is being accessed by a scraper or a human.

Maybe we should just ban User Agent strings and be done with it.

It really doesn't take much effort to detect the majority of scrapers. Usually you do so by monitoring patterns of any given IP.

Is each request a profile page incremented (/users/1, /users/2, etc)

or dozens of requests a minute (faster than a typical user would read)?

Is static content (particularly images and CSS) being downloaded too or just the HTML content?

Sometimes the referrer HTTP header can give clues too - though you have to be careful there as that's as unreliable as the user agent header.

However if you're really paranoid about scrapers you can also throw in some honeypots. eg a fake user (/users/13) which is a user account that doesn't exist so that page wouldn't have any links from within your site. ie you only reach it if you're incrementing through the user IDs. Or perhaps a link within your HTML which doesn't render so it's only reachable via automated scripts that don't check what links are rendered inside the display view. Anyone that gets ensnared in your honeypot could then be put on a temporary IP blacklist. Though the danger of doing this is you accidentally blacklist good crawlers if you're not careful about setting appropriate robots rules.

I'd be willing to bet that the user-agent field isn't the problem; it's patterns that everything looks for now, right? People have been lying in the request headers for decades.

Yeah, I remember a plugin in the early days of Firefox that allowed Firefox pretend to be Internet Explorer. It would send request headers that made it appear to be IE.

There were websites at the time that would display just fine in Firefox, but would refuse to display anything if they detected a non-IE browser.

I've kind of always thought that we shouldn't be using UA strings. Just give the requester the data that they requested according to the current open standards. If they choose not to render it correctly, then thats their problem.

Yes, I realize that it's not that simple, but I think browsers would have tried much harder to adhere to standards if we had done it that way.

You call it the "public internet" but it's most definitely not a public space or anything like it.

Private entities own and operate all(most of) the servers, services and conduits, and that does need to be paid for and maintained.

I'm not saying I agree with Linkedin in this particular scenario, but this is about two commercial for-profit entities arguing over money, so let's not make it about something it's not.

> Private entities own and operate all(most of) the servers, services and conduits, and that does need to be paid for and maintained

And are MORE than happy to send the content of their servers to unsolicited, uninvited, anonymous guests on mere request. No-one is forcing them to do so!

And that's exactly LinkedIn's position here.

No one should be forcing them to send their content to anyone.

They claim they should be allowed to discriminate at their discretion who they respond to, since they own and operate the servers.

This "no one is forcing you to send your content" goes both ways..

If one side is going to say they're entitled to receive the content on request, the other side wants to be able to say they're entitled to refuse to answer that request..

>"You call it the "public internet" but it's most definitely not a public space or anything like it."

How is it not a "pubic space"? They publish publicly visible A records for their site as well as route their public IP space to transit providers in order for the public to be able to reach their site.

If I publicly distribute posters about a party in my apartment and leave the door unlocked, do I still have the right to kick out anyone I don't like?

That's a pretty diengenuous strawman.

A more accurate comparison would be that you put up an advertisement on a billboard along a busy street and then decided to tell people who passed by that they weren't allowed to take a picture of it.

And to continue with this absurdity you feel entitled to enforce who can or can not look at your billboard because despite it being publicly viewable its your advertisement on the billboard.

That's also not accurate.

There is no "public space" on the Internet.. There's no un-owned territory or resource that is free to use or metaphorically "stand around" in to take those pictures from.

You are consuming privately-owned resources in all your online activities, and as such some will argue that they can decide to limit your consumption of those resources at their own discretion.

Again, I am not siding with either party here, just trying to dispel this notion that "public space" - in the way we understand public space to exist in the physical world - exists on the internet.

In your example, no one is controlling your right to take photos or stand around and look in any direction you choose.

When you use the Internet, a private entity is allowing you to transit through their network and access sites, a different entity is allowing you to access and receive their content, etc..

LinkedIn owns the server you are accessing when you (or others) go to their site, and they are spending resources servicing those requests, and - they claim - can decide how and when they choose to do that..

> Being a programmer not a lawyer, I like the idea of more rights for scrapers.

What rights should scrapers have that they don't right now? Keep in mind that a lot of the scraping going on is just some other private company abusing access and hoping to gather and use the information for their own private profit. How many companies are scraping StackOverflow for example and doing nothing but attempting to copy it and draw traffic to their own site? I can't stand copycat sites, they fill my search results with junk. I would assume the majority of scraping that is currently happening is not doing the public any good.

> I don't want to see the internet partitioned away and owned by a few companies, especially when that information is often called a "public profile".

This sounds like you're suggesting that LinkedIn or Facebook calling your profile a 'public profile' means that the law should treat it as a public service due to use of the word 'public', is that what you mean? The word public may be overloaded here. I can see why tax funded projects should be publicly accessible, but I have a hard time seeing why private companies should be compelled to provide access to anything at their own expense.

Consider that should anything ever happen to the sites they scrape, suddenly they become super valuable to the rest of us. Decentralized information is not a bad thing. And arguably, if monetizing those scraped sites pay for them duplicating the data to additional places, I think that's probably fine.

Arguably, if they do a better job getting that information in results to people who need it in search, they may be performing a service there as well. (A lot of decently informative sites have absolutely awful search/visibility.)

> Consider that should anything ever happen to the sites they scrape, suddenly they become super valuable to the rest of us.

That may well be true, but that value doesn't mean anyone should just be able to take that value from the company that put up the effort and investment to collect the data, and turn around an use it for their own profit. Nor does it mean that a company shouldn't be able to serve the data to whomever it wants and/or restrict access from whomever it wants. Value to the consumer is still not a reason to compel private companies to offer public services. It would be valuable to both of us if Google gave us free money, but no court is going to compel them to do so just because of the potential value to you and me.

It seems bad, btw, if we choose to rely on private companies to keep the only backups of our personal data. If a site going down has a negative effect on my life, and takes down data with it that I need, it might be an indication that I shouldn't have kept my data there.

Also true that decentralized information is not a bad thing, as a generic ideal or a data backup plan. But for a business, decentralization in this context means loss of profit, as well as possibly theft, cheating, and copyright violation.

Also, the increased value that comes from a company folding will be used against you by these private, for profit scrapers. They can and will hold their copy ransom for more money, if possible.

> Arguably, if they do a better job getting that information in results to people who need it in search, they may be performing a service there as well. (A lot of decently informative sites have absolutely awful search/visibility.)

What is the argument in favor of this being legal? It is currently not legal, and the law currently does not give any credit for 'doing it better'. Why should it?

It seems like a really dangerous precedent to interpret "unauthorized access" in CFAA to mean accessing a computer that is made available publicly without any form of access control when the owner doesn't want you to access it instead of meaning that you subverted some form of access control to access the computer.

I'd argue if the data is being made publicly available (particularly in the case of LinkedIn where we are choosing to make our own data publicly available), it is being made publicly available, and anyone should be able to take and use that data. Our data shouldn't be what a corporation's profit (or loss of profit) is based on to begin with.

> if the data is being made publicly available ... anyone should be able to take and use that data

It isn't being made publicly available in that sense. LinkedIn only offers the data to site visitors (unregistered users) under the guise of a license.

> we are choosing to make our own data publicly available

This isn't true. Putting data on LinkedIn is not making it publicly available, it's sharing a copy of your data with LinkedIn, and allowing them to do whatever they want with it. Those are the terms you agree to when you register.

> Our data shouldn't be what a corporation's profit (or loss of profit) is based on to begin with.

I agree, in an ideal world, but LinkedIn does profit on your data (as do Facebook, Google, Microsoft, etc.). And we are willingly sharing our data with them and allowing this to happen. There are all kinds of crappy trends with data and privacy happening, and lots of people raising red flags. Your choice is to not use those services. If you don't want LinkedIn to use your data for their profit, then don't share your data with LinkedIn. If you share your data with LinkedIn, then LinkedIn now has the right to use your data to their own advantage.

Can they claim a tax credit for supporting that bandwidth usage and handling abuse?

Not needed. Cost of doing business.

I don't trust the US government to write good rights for scrapers. They can't even do computer crime sentences well.

At best, it's a burden for no solid gain for society. At worst, there will be loopholes used to DoS businesses because they can't shut down individuals due to law-given rights, and that will lead to court fights.

These rights would do nothing but save scraper authors from learning to obfuscate their actions.

Information wants to be free. People should stop fighting it!

If one makes information "public" but don't really want to share it, then the public is fully justified in taking it.

<< Information wants to be free. People should stop fighting it!

That's why you routinely publish your bank account, social insurance, and credit card details online, right?

Not like I have a choice... and how are those things explicitly declared 'public'?

The way I see it, if you want to make a 'public' profile, host it yourself, create your own website. LinkedIn has spent a lot of resources to create a service/network that helps to connect you with recruiters (or the opposite), whose information may not be easily available to you or can be found easily in the 'public' Internet. If you think from a product owner's standpoint, why should I let just anyone scrape the content collection that I built with so much effort?

Agree. However, I think there is a difference between harvesting a database's contents and observing trends, data and creating algorithms based around that observed data.