News for hackers. Stuff that matters.
Re:How Google’s Web Crawler Bypasses Paywalls by zaroth on Friday February 19, 2016 @02:03PM (#11135701)
And congratulations, you have likely just "exceeded authorized access" and committed a felony violation of the CFAA punishable by a fine or imprisonment for not more than 5 years under 18 U.S.C. § 1030(c)(2)(B)(i).

From the ABA: "Exceeds authorized access is defined in the Computer Fraud and Abuse Act (CFAA) to mean "to access a computer with authorization and to use such access to obtain or alter information in the computer that the accesser is not entitled so to obtain or alter."

To prove you have committed this terrible felony, the FBI will now demand that Apple assist in disabling the secure enclave of your device in order to access your browser history. But remember, they only need to do this because they aren't allow to MITM all TLS and "acquire" -- not "collect" -- every HTTP request your machine ever makes. </s>

Re:How Google’s Web Crawler Bypasses Paywalls by jrockway on Friday February 19, 2016 @02:22PM (#11135862)
User agent strings have a long history of being intentionally misleading. IE 11 claims to be "Mozilla/5.0". Chrome claims to be "Safari/537.36". The User-Agent string is all lies, and has been ever since the first site started doing UA sniffing.
Re:How Google’s Web Crawler Bypasses Paywalls by zaroth on Friday February 19, 2016 @02:27PM (#11135902)
It's intent that matters. Setting user-agent in order to properly render a page is legal. Setting a user-agent string to gain access to otherwise unauthorized content is probably not.
Re:How Google’s Web Crawler Bypasses Paywalls by dheera on Friday February 19, 2016 @02:55PM (#11136127)
In that case, I'm just going to set my User-Agent permanently to Google's crawler for fun and see how the web renders, in general. Just for fun, because I want to see what the world looks like from a different perspective. That's my intent. I've stated it. Now that my user agent has been changed, I'm going to go grab lunch, carry on with life, maybe check out some cat pictures and then maybe some news sites over tea and snacks, and then go for a run.
Re:How Google’s Web Crawler Bypasses Paywalls by lojack on Friday February 19, 2016 @03:39PM (#11136448)
In my opinion the WSJ has a major rendering bug that affects everyone except for the google search crawler, and I selectively set my User Agent to get around said bug.
Re:How Google’s Web Crawler Bypasses Paywalls by comex on Friday February 19, 2016 @04:03PM (#11136607)
In any judge's opinion, you'd be full of shit. Judges aren't stupid.

Which doesn't matter, because the WSJ is never going to sue you. But make sure you only consider your justification a personal one, not one that would provide any legal protection.

Re:How Google’s Web Crawler Bypasses Paywalls by djsumdog on Friday February 19, 2016 @05:10PM (#11137053)
Most judges and legislators are fucking morons. Legal systems aren't a science. I wrote a thing on it a few years ago:

http://khanism.org/security/legality/

Re:How Google’s Web Crawler Bypasses Paywalls by newjersey on Friday February 19, 2016 @05:29PM (#11137198)
I thought the problem was that it doesn't matter if WSJ is malicious in this case. Because cfaa is a criminal matter, zealous police departments and district attorney's offices can pursue heavy handed cases even without much cooperation from "the victim" which in this case is a corporation.

All I want to say is scrape cfaa now!

Re:How Google’s Web Crawler Bypasses Paywalls by bigjust on Friday February 19, 2016 @06:53PM (#11137696)
> scrape cfaa now!

pun?

Re:How Google’s Web Crawler Bypasses Paywalls by profeta on Friday February 19, 2016 @02:36PM (#11135976)
intent can be many things.

i disable javascript. so i can't even see their page (and hence i don't know my access is being denied since i got a 200 http response, which means "OK") so i try different user agents with the intent of reading the content they are providing. Just like microsoft case.

Re:How Google’s Web Crawler Bypasses Paywalls by CyberDildonics on Friday February 19, 2016 @03:55PM (#11136547)
Unauthorized? If you put up a sign and tell people they have to pay to look at it, is it illegal to look at it and not pay? This should be a rhetorical question.
Re:How Google’s Web Crawler Bypasses Paywalls by sneak on Friday February 19, 2016 @04:20PM (#11136712)
[I am not a lawyer and this is not legal advice.]

It is not specifically defined in the law, so it reverts to the traditional meaning: anything the owner of the system says you aren't authorized to access.

It's lunacy, I know. That's what HTTP headers and WAFs and such are for. But that's the stupid law, and it sent someone who used to be my friend to federal prison for changing a user agent and referrer and accessing unprotected data on the web.

Tread carefully.

Re:How Google’s Web Crawler Bypasses Paywalls by tomcam on Friday February 19, 2016 @05:03PM (#11136993)
Whoa. Care to tell that story or link to it?
Re:How Google’s Web Crawler Bypasses Paywalls by lvs on Friday February 19, 2016 @05:44PM (#11137308)
I believe this refers to the Weev case.
Re:How Google’s Web Crawler Bypasses Paywalls by cookiecaper on Friday February 19, 2016 @06:22PM (#11137535)
It may, but weev is probably not the only person whose been put away for that. This type of activity is the basis for Google and many other tech startups. I hope they catch Larry and Sergei soon, they've been on the lam for almost two decades!
Re:How Google’s Web Crawler Bypasses Paywalls by piyush_soni on Saturday February 20, 2016 @04:52AM (#11139561)
But that one is for "greater public good" you see .... ;)
Re:How Google’s Web Crawler Bypasses Paywalls by sbierwagen on Friday February 19, 2016 @04:10PM (#11136648)
18 USC § 1030 (a) (4) https://www.law.cornell.edu/uscode/text/18/1030

  Knowingly and with intent to defraud, accesses a 
  protected computer without authorization, or exceeds 
  authorized access, and by means of such conduct 
  furthers the intended fraud and obtains anything of value,
The courts have interpreted "protected computer" as any computer connected to the internet.
Re:How Google’s Web Crawler Bypasses Paywalls by majewsky on Saturday February 20, 2016 @06:09PM (#11142422)
> The courts have interpreted "protected computer" as any computer connected to the internet.

The mismatch between the world views of jurisprudence and engineers is a neverending source of joy. (If working in tech has made you cynical like me, that is.)

Re:How Google’s Web Crawler Bypasses Paywalls by jsprogrammer on Friday February 19, 2016 @02:39PM (#11136001)
User-agent string is not an authorization mechanism.
Re:How Google’s Web Crawler Bypasses Paywalls by zaroth on Friday February 19, 2016 @02:50PM (#11136085)
Interestingly, the CFAA does not define the term "without authorization" however it does define "exceeding authorization" exactly as I quoted above;

  - to access a computer with authorization and to use such access
    to obtain or alter information in the computer that the accesser
    is not entitled so to obtain or alter.
So arguing User-agent is not an authorization mechanism probably won't help you, because exceeding authorization means, first, that you were authorized to access the computer (HTTP GET returns 200) but then that you used that access to obtain information in the computer that you were "not entitled so to obtain."
Re:How Google’s Web Crawler Bypasses Paywalls by jsprogrammer on Friday February 19, 2016 @02:55PM (#11136122)
If the computer on the public Internet responds to a standard HTTP request, it has explicitly authorized my access to whatever information it sent me.
Re:How Google’s Web Crawler Bypasses Paywalls by zaroth on Friday February 19, 2016 @03:05PM (#11136203)
Unfortunately this will not help your defense. Andrew "Weev" Auernheimer was convicted of violating CFAA for exactly this (although the conviction was later overturned on a technicality).

Again, exceeding authorized access means using your authorized access to obtain information you were not "entitled" to. So the question is not 'were you authorized' but rather it is 'were you entitled' to that information? WTF 'entitled' means is another question entirely, but likely it is in the eye of the beholder. A jury decided Weev was not 'entitled' to the email addresses he downloaded from AT&T, and it's safe to assume we are not 'entitled' to free access to WSJ's content. So I would not rest your hopes on the "200 OK".

Re:How Google’s Web Crawler Bypasses Paywalls by jsprogrammer on Friday February 19, 2016 @06:27PM (#11137563)
> Andrew "Weev" Auernheimer was convicted of violating CFAA for exactly this (although the conviction was later overturned on a technicality).

So, your example is...not an example?

>WTF 'entitled' means is another question entirely

No, in this case it is very clear: a request containing a particular user agent string is entitled. I have not tried this myself, but presumably you could verify that is the case by sending a request with the appropriate user agent.

Re:How Google’s Web Crawler Bypasses Paywalls by zaroth on Friday February 19, 2016 @07:06PM (#11137745)
It's the best example we got. The case was overturned (after he spent quite some time in federal prison) not because it was found that he didn't violate the CFAA but because the charges were brought in the wrong jurisdiction.

Again I think you're confusing the fact someone could trick the server into delivering the content for free with WSJ intending to deliver their content to you for free. Since WSJ clearly intends their content to be delivered to only Googlebot for free and to users only if they pay, it is likely a jury would consider this a violation of CFAA.

A web server returning 200 OK is not ipso facto a guarantee the person making the request is not committing a crime. To give a more obvious example, if the request header contains a stolen authorization token. The law does not require the access control be non-trivial to defeat.

I don't like it, and I think the CFAA is seriously problematic, but it is the law and the Feds have been known to enforce it.

Re:How Google’s Web Crawler Bypasses Paywalls by jsprogrammer on Friday February 19, 2016 @09:23PM (#11138369)
As far as the law is concerned, he did not violate anything. You do not have to prove yourself innocent, the burden is on the prosecutor to prove a violation. In the example you cite, no violation has been shown.

It is not at all clear that WSJ intends Googlebot to get their content for free while others must pay. Thus is actually against Google's policies, which would actually call into question whether WSJ's behavior is felonius. WSJ may not be entitled to be incorporated into Google's index, yet they are manipulating the Googlebot to the contrary.

Re:How Google’s Web Crawler Bypasses Paywalls by jsprogrammer on Saturday February 20, 2016 @11:38AM (#11140744)
If you will down mod, at least show how I am wrong!
Re:How Google’s Web Crawler Bypasses Paywalls by geofft on Friday February 19, 2016 @08:49PM (#11138249)
> So, your example is...not an example?

At least in the US, the law doesn't work that way. Decisions will quite often cite some other similar case which reached the opposite conclusion, but under different circumstances, because that other case's decision says something like "X, if it weren't for Y" or "Fortunately for the defendant, they didn't Z, so not X", or something. That isn't binding precedent for the judge to apply X, but it's a very strong sign that X would be reasonable.

A court case that says "Yes, this violates CFAA but we have to throw out the case because A, B, and C" is very strong reason to believe that, if the next prosecutors avoid A, B, and C, the next judge will say "Yes, this still violates CFAA."

(IANAL but I read court cases because I find it useful to understand my jurisdiction's legal system.)

> a request containing a particular user agent string is entitled.

The phrasing of the law is very clear that the word "entitled" applies to a person, not to a request. Stealing someone's password and using their account is definitely a violation of CFAA (see e.g. http://www.wiggin.com/16332). In such a case, the account used to log in is quite plainly "entitled" / "authorized;" that's how you get the data. But the person logging in is not "entitled".

Re:How Google’s Web Crawler Bypasses Paywalls by jsprogrammer on Friday February 19, 2016 @09:16PM (#11138341)
We aren't talking about user names and passwords, but user agent strings.

The other decision was vacated. The jury was not appropriate and their decision is irrelevant.

Re:How Google’s Web Crawler Bypasses Paywalls by compiler-guy on Friday February 19, 2016 @03:22PM (#11136333)
You are confusing an engineering process with a legal one.

Unfortunately, when the prosecutors come, they will only care about the legal process.

Re:How Google’s Web Crawler Bypasses Paywalls by CyberDildonics on Friday February 19, 2016 @03:57PM (#11136556)
It's not engineering, it's common sense.
Re:How Google’s Web Crawler Bypasses Paywalls by compiler-guy on Friday February 19, 2016 @04:19PM (#11136706)
Common sense is not a set of legal procedures and rules either.

The legal world cares about how the law applies to the facts of the case, not about how common sense applies.

Not saying I like it.

Re:How Google’s Web Crawler Bypasses Paywalls by jsprogrammer on Friday February 19, 2016 @06:33PM (#11137613)
The facts are dictated by the engineering. Is a lawyer a computer networks expert? Not by default. They will need to defer to the engineers.
Re:How Google’s Web Crawler Bypasses Paywalls by compiler-guy on Friday February 19, 2016 @07:13PM (#11137784)
Certainly, some of the facts are dictated by the engineering.

However, the set of people "authorized" is not, at least not from a legal perspective. This is what the case law says. The fact that the set of people who technically _can_ access the data is different from the set of people legally authorized to access the data.

That might not be what the engineers who designed the system, run it, and produce the content intended, but that is what the law says.

It's a bummer the two disagree. But only one of the two systems put you in jail if you cross them.

You and I may wish it were otherwise, but wishing isn't going to make it so.

Re:How Google’s Web Crawler Bypasses Paywalls by Spare_account on Friday February 19, 2016 @03:07PM (#11136216)
Your request wouldn't be 'standard', it would be deliberately malformed to bypass a paywall. You guys are performing linguistic gymnastics to get around that fact.
Re:How Google’s Web Crawler Bypasses Paywalls by oldmanjay on Friday February 19, 2016 @03:15PM (#11136288)
As you've indicated, a request can only be considered malformed if it doesn't conform to the standards. Here is what the relevant RFC[0] has to say about the User-Agent header:

> Likewise, implementations are encouraged not to use the product tokens of other implementations in order to declare compatibility with them, as this circumvents the purpose of the field. If a user agent masquerades as a different user agent, recipients can assume that the user intentionally desires to see responses tailored for that identified user agent, even if they might not work as well for the actual user agent being used.

That sure sounds like impersonating other user agents is allowed, but not encouraged. That is a clear distinction from being malformed.

[0] https://tools.ietf.org/html/rfc7231#section-5.5.3

Re:How Google’s Web Crawler Bypasses Paywalls by Spare_account on Friday February 19, 2016 @03:25PM (#11136359)
Unless you were being tongue-in-cheek, you've completely sidestepped the intent of my comment and continued with the linguistic silliness.

Obtaining paywall-protected content by faking your user agent to purport yourself to be a Google Crawler is quite clearly fraudulent. This isn't a point for debate.

PS. To play along with the linguistic theme, can you provide a source for the definition of a malformed request? My original intent when using the word malformed was not to invoke it's technical definition but rather it's dictionary definition. But, having said that, I just had a 30 second Google hunt and couldn't find anything to corroborate your position.

Re:How Google’s Web Crawler Bypasses Paywalls by jsprogrammer on Friday February 19, 2016 @04:55PM (#11136937)
So...if I only entitle Mozilla based agents to view my content and Chrome, IE, and Safari users still get access by claiming to be a Mozilla agent, those users have committed a felony?

This line of argumentation is beyond ridiculous.

Re:How Google’s Web Crawler Bypasses Paywalls by Spare_account on Friday February 19, 2016 @05:15PM (#11137078)
Are your Mozilla users paying for access? Are the Chrome users with the fake agent strings obtaining paid-for content for free? The use of forged agent strings isn't the offence.
Re:How Google’s Web Crawler Bypasses Paywalls by oldmanjay on Friday February 19, 2016 @03:48PM (#11136505)
I wasn't playing a linguistic game. You mentioned standards and malformed requests, and I pointed out that you misused those terms. I am not bothering to track down definitions for you to play, as you say, linguistic games. You are free to find something that proves my position wrong, as I used the RFC to cite my position as correct.

If you meant to put forth that faking a user agent is a technique to exceed authorization, that's fine, and I'm glad to have helped you clarify it. Just be clear, it's not what you said with your detour into malformed requests.

Re:How Google’s Web Crawler Bypasses Paywalls by mcherm on Friday February 19, 2016 @04:13PM (#11136662)
> a request can only be considered malformed if it doesn't conform to the standards

That is demonstrably false. I, personally, can consider any request malformed unless it starts with the letter W. Regardless of what the standards say, I can think whatever I want to.

Similarly, the law can make up whatever rules IT wants to about the definition of "malformed". In that case, it pays some scant attention to things like standards, but mostly cares about "to the random guy-on-the-street (jury member or judge) did this seem like stealing". And there, I am afraid you lose.

Re:How Google’s Web Crawler Bypasses Paywalls by TheDong on Friday February 19, 2016 @05:01PM (#11136975)
That argument doesn't matter thought. He didn't say "a request can only be considered malformed by mcherm's definition of the term if...".

I could say "the sky is blue" and you could say "demonstrably false; the sky is personally red because I'm wearing colour-altering sunglasses".

What he meant is obviously "a request can only be considered malformed (as defined by the commonly accepted definition of malformed) .."

Your argument is fallacious. The courts will use common definitions of terms and have regard for context.

However, what you're arguing about is even more pointless. The CFAA doesn't depend on the term "malformed" in any way, but on the term "authorized access".

Your first sentence is not similar at all to your second; they're demonstrating first a blatant disregard for common definitions of terms, followed by praise of such common definitions.

Re:How Google’s Web Crawler Bypasses Paywalls by ignoramous on Friday February 19, 2016 @02:48PM (#11136062)
It isn't. But I have seen top internet companies (Netflix and the like) use it to authorise requests. It boggles my mind.
Re:How Google’s Web Crawler Bypasses Paywalls by seanp2k2 on Friday February 19, 2016 @05:52PM (#11137364)
Curious where Netflix is using this to authorize requests. Any more info on that?
Re:How Google’s Web Crawler Bypasses Paywalls by ignoramous on Friday February 19, 2016 @08:14PM (#11138100)
Their Android app wouldn't let you through if the device is unrecognised, that is, if your user agent has strings which Netflix hasn't whitelisted. They also check model ID, and build fingerprint too, I believe. A standard practice in the Android world, for some unknown reason. I don't know if they have stopped doing that.
Re:How Google’s Web Crawler Bypasses Paywalls by njharman on Friday February 19, 2016 @03:12PM (#11136254)
Good luck arguing that in court. Esp against the multiple lawyers a corporation will be able to afford.

If you don't get it, court doesn't care about what's reasonableness, technically correctness, etc. Only if your lawyers can convince jury/judge. Twinky made me crazy, It the gloves don't fit... and so forth.

Re:How Google’s Web Crawler Bypasses Paywalls by tajen on Friday February 19, 2016 @05:48PM (#11137333)
If you don't set it for a particular website but generally browse the web with it, based on another legitimate purpose (I'm a developer, I had to test a website, I forgot the setting) could you oppose the court on "knowingly and with intent of defraud"? If you didn't see the paywall, how can it be "knowingly"?
Re:How Google’s Web Crawler Bypasses Paywalls by njharman on Saturday February 20, 2016 @11:17AM (#11140655)
Trot out expert witnesses (on web dev). "Sir, have you ever "forgot" the setting?" No, and it's ridculous to think anyone would. "Is it generally know by web devs the dangers and circumvention ability of this setting" Absolutely. etc.

"Here is a transcript of electronic forum detailing how to circumvent access controls and defraud the victim using the exact methods defendant used to access victim's website. A forum the defendent heavily traffics. Often multiple times per day."

Ladies and gentlmen I ask you is it more likely that the defendent, a self professed developer, and expert in these circumvention methods, who reguralry participates on forums discussing hacking and defrauding companies such as the victim. I ask you is it reasonable to believe he "just forgot"?

Lawyers man, Lawyers! Can you not understand that rationality, technicallity don't matter. Lawyering is like statistics/graphs. You can get the data to say whatever you want.

Re:How Google’s Web Crawler Bypasses Paywalls by jsprogrammer on Saturday February 20, 2016 @02:19PM (#11141459)
User agent string is not an access control mechanism.
Re:How Google’s Web Crawler Bypasses Paywalls by antsar on Friday February 19, 2016 @02:43PM (#11136034)
A decent Friday afternoon read on that topic:

http://webaim.org/blog/user-agent-string-history/

Re:How Google’s Web Crawler Bypasses Paywalls by jd3 on Friday February 19, 2016 @07:29PM (#11137873)
I couldn't believe my eyes when I saw that netflix started sniffing the UA. You can't even play a video in SeaMonkey 2.39 without the incredibly stupid

general.useragent.override.netflix.com;Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:42.0) Gecko/20100101 Firefox/42.0

in about:config... what were they thinking?

Re:How Google’s Web Crawler Bypasses Paywalls by dpweb on Friday February 19, 2016 @03:52PM (#11136531)
Pretty obvious weak move restricting by UA$.

Allow by IP range? You can probably find a somewhat accurate range for Google and whoever's crawlers.

Re:How Google’s Web Crawler Bypasses Paywalls by cookiecaper on Friday February 19, 2016 @04:02PM (#11136606)
Google's _entire business_ is based not only on accessing servers without authorization, frequently in explicit violation of that site's terms of use because ToU boilerplate includes languages excluding all "crawlers, bots, or spiders", but also on flagrant violation of copyright law. They save the entire text of the web page on their servers when they crawl it (unlicensed copying), rehost it in Google Cache (unlicensed redistribution), save all the images and rehost them on Google Images (both), and so forth. All of this is absolutely illegal under current copyright law.

Google was fortunately that no one sued them for these things before they got big enough to defend themselves. Many tech entrepreneurs haven't been so lucky.

You may ask why a big company like Google isn't doing more to change the CFAA or copyright law. The reason is now that they're big enough, legal grey areas like those in the CFAA (particularly, "what is unauthorized access?", because it's not defined by the statute) can be fully exploited, and Google can sit secure in the knowledge that they'll never be realistically challenged on it; meanwhile, they can then threaten potential competitors for doing the same thing, since a lawsuit against a public corporation takes 10 years and $5MM-$20MM. Anyone who could mount that kind of offense against Google won't, because they benefit from the grey area too; they'll just make some backroom deal with Google and not lose their lucrative, competition-destroying ability to do things that companies with sub-$100M revenues aren't able to do.

Re:How Google’s Web Crawler Bypasses Paywalls by roywiggins on Saturday February 20, 2016 @12:57AM (#11139097)
If those sites don't want spiders, they can just specify that in robots.txt, which Google honors, right?

From the point of view of the law, that might not matter, but when there's a standardized way to make clear to bots that they're not welcome and you didn't bother to implement it, you'll look pretty silly if you complain.

Re:How Google’s Web Crawler Bypasses Paywalls by dosaygo on Saturday February 20, 2016 @12:56AM (#11139095)
Okay, there are other aspects to Google's business beside information retrieval and search, tho the point that the access is mostly unauthorized is valid. Although the damage done by this is to some extent offset by Google's status essentially as a public utility: it universally provides a social good, "search", and the tax we pay is advertising. Your frustration for the "little guys" is mostly en pointe, since the damage done is rendered then mostly to competitors, rather than to customers, and the argument, true or no, that Google's search is "lightyears" ahead of other possible offerings, to some extent offsets the damage to consumers due to loss of competition through the possibly anti-competitive practices you highlight. So it seems the balance of good is in the favor of consumers of "search". I think this is the main force, rather than any "structural obstacles" to competition, which is the cause of the persistence of the status quo in this market. The thing about this which no one seems to see is that, since everyone is taking information dishonestly, there's a huge opportunity to actually "bring it into the light", do it honestly, and strike some kind of deals with content creators.
Re:How Google’s Web Crawler Bypasses Paywalls by skissane on Friday February 19, 2016 @04:34PM (#11136805)
Doesn't this just prove that CFAA (and similar laws in other jurisdictions) is massively over-broad and needs to be narrowed. I'm surprised there is not more campaigning for CFAA repeal and replacement with far narrower legislation.

If someone hasn't exploited a security bug (I mean a real security bug, like a buffer overflow, I don't consider behaviour such as serving up content to certain User-Agents only a genuine "security bug"), and they haven't bruteforced/cracked/acquired a password or private key, and they aren't sending unreasonable amounts of traffic ((D)DOS), it should not be a crime, and the law should be changed to reflect that principle. The law should reflect the common sense of the technically literate, but it doesn't, because it was written by the technically illiterate.

Re:How Google’s Web Crawler Bypasses Paywalls by swanson on Friday February 19, 2016 @02:17PM (#11135829)
I wonder if a user-agent that was something like "Not a Googlebot" would a) allow access (probably regex based) and b) be truthful/plausible deniability.
Re:How Google’s Web Crawler Bypasses Paywalls by oneeyedpigeon on Friday February 19, 2016 @05:43PM (#11137292)
What is your purpose in setting your user-agent string to that value, other than cleverly bypassing the paywall?
Re:How Google’s Web Crawler Bypasses Paywalls by DanielBMarkham on Friday February 19, 2016 @05:52PM (#11137361)
As this entire thread has pointed out, the law makes no sense at all here. So you might as well say "Congratulations! You're going to jail! And I don't have to tell you why!" because none of the details matter anyway.

This is the spot we've reached through legislative meddling, and your best bet is either to be a good little consumer and don't make waves or do what you want and don't get caught. Neither of those seem to make a lot of sense in the long run.

So yes, congrats. It's jail for you -- but not for Google. Because they're Google, and you, well, you're not. You'd just better be happy we don't find out about you rooting your cellphone last year. Good grief.

ADD: I know you meant well, and I appreciated the </s> tag, but there was something I didn't like about your comment. Now I know what it is. By making this a big deal, you're increasing the likelihood that this poor schmuck becomes the next "example" some federal prosecutor decides to make. It's not your fault, but it still sucks. Let's hope that doesn't happen.

Re:How Google’s Web Crawler Bypasses Paywalls by pbosko on Friday February 19, 2016 @03:55PM (#11136548)
How does that law apply to foreigners?
Re:How Google’s Web Crawler Bypasses Paywalls by ec109685 on Friday February 19, 2016 @02:06PM (#11135725)
Who has? Surely not the person who wrote the tutorial.
Re:How Google’s Web Crawler Bypasses Paywalls by zaroth on Friday February 19, 2016 @02:11PM (#11135770)
Even worse, the poor author has created a hacking tool capable of enabling said felony, which I believe could get them 10 or 20 years... I'm looking for the statute now.

Edit: I was mis-remembering, the current law is against possession or manufacture of eavesdropping or wiretapping devices, not hacking tools. The EU has been playing with laws against hacking tools, but apparently nothing in the US yet against it.

The law makes it illegal to distribute devices (incl. software) that the design of such [software] renders it primarily useful for the purpose of the surreptitious interception of wire, oral, or electronic communications. Punishable by not more than 5 years and/or not more than $250,000. 18 U.S.C. 2512.

I don't think this blog post qualifies as an "interception" device,... however unauthorized retrieval and recording of another's voice mail messages constitutes an "interception" so who the hell knows. I'm sure you could find a US DA who would argue the falsified User-Agent meant the software is designed to "intercept" communication meant only for Google.

Re:How Google’s Web Crawler Bypasses Paywalls by majewsky on Saturday February 20, 2016 @06:28PM (#11142474)
The German Criminal Code has §202c StGB [1], commonly known as the "hacker section" (or "Hackerparagraph" in German).

While §202a and §202b punish unauthorized access to or interception of data, §202c extends this threat to obtaining passwords or creating tools in preparation of such an act.

Note that this section has only existed for a few years. I am not aware if white-hat hackers have actually been prosecuted for creating hacker tools.

[1] http://www.gesetze-im-internet.de/stgb/__202c.html

Re:How Google’s Web Crawler Bypasses Paywalls by stonemetal on Friday February 19, 2016 @05:55PM (#11137382)
Wouldn't it run afoul of the DMCA for making available a tool for circumvention?
Re:How Google’s Web Crawler Bypasses Paywalls by jerf on Friday February 19, 2016 @02:11PM (#11135765)
Under CFAA, I don't know. The DMCA may have some problems with that blog post, though.

And by "may", I do mean "may". I don't know. But it's at least possible.

Re:How Google’s Web Crawler Bypasses Paywalls by zaroth on Friday February 19, 2016 @02:40PM (#11136007)
That's a great point, it's likely an illegal DMCA circumvention device too!

  No person shall manufacture, import, offer to the public, provide, or otherwise traffic
  in any technology, product, service, device, component, or part thereof, that—

  (A) is primarily designed or produced for the purpose of circumventing a technological
      measure that effectively controls access to a work protected under this title;

  (B) has only limited commercially significant purpose or use other than to circumvent a 
      technological measure that effectively controls access to a work protected under
      this title; or

  (C) is marketed by that person or another acting in concert with that person with that
      person’s knowledge for use in circumventing a technological measure that effectively
      controls access to a work protected under this title.
I mean, obviously it's all quite ridiculous, but also deadly serious at the same time :-(
Re:How Google’s Web Crawler Bypasses Paywalls by majewsky on Saturday February 20, 2016 @06:30PM (#11142481)
> is primarily designed or produced for the purpose of circumventing a technological measure that effectively controls access to a work protected under this title

The wording reminds me of a similar section in the German Copyright Law which outlaws the circumvention of "effective" copy-protection schemes.

How an access control scheme can be effective and circumventable at the same time is completely beyond me. :)

Re:How Google’s Web Crawler Bypasses Paywalls by 1024core on Friday February 19, 2016 @05:55PM (#11137381)
I know you were kidding, but I still don't understand this "exceeds access" bullshit.

Here's how HTTP(S) works: I issue a REQUEST to the web server; the web RESPONDS with it, or denies it. It is up to the web server to respond or deny or do whatever it wants. If the web server is badly implemented or doesn't know what it's doing, it is the webserver's fault.

Remember: it's just a request. I can request 100 dollars from you; the fact that you give them to me does not make me a mugger.

Re:How Google’s Web Crawler Bypasses Paywalls by barrkel on Friday February 19, 2016 @06:05PM (#11137445)
What if you tell me my car's broken when it's not and request 100 dollars from me to fix it? That would be fraud.

So you go up to a server and lie to it, and it gives you something; is that not acquiring things through deception?

The structure of your argument suggests that e.g. breaking into an ssh server by issuing a login request with a known password which it responds to, isn't illegal. And further, that if data is acquired from the server, there is still no crime - the ssh protocol too is just requests to the server, it's all bits down the line. It's clearly nonsense.

Re:How Google’s Web Crawler Bypasses Paywalls by maxerickson on Friday February 19, 2016 @06:02PM (#11137422)
If you request the 100 dollars with a specially crafted piece of paper that I glance at and believe is legitimate, you are committing a crime (check fraud). Which whatever, let's not focus on the analogy. Whether you like it or not, the law doesn't necessarily view the valid server response as authorization to access the requested url, it examines what you were thinking when you created the url.
Re:How Google’s Web Crawler Bypasses Paywalls by prebrov on Friday February 19, 2016 @08:45PM (#11138229)
And one more car analogy. If someone comes up to your car, pulls the door (issues a request) and based on the fact that it opens (responds 200), drives away in it, that would be seen as grand theft auto.

Whether the lock was implemented poorly or you just didn't lock it — doesn't matter.

Re:How Google’s Web Crawler Bypasses Paywalls by fiatmoney on Friday February 19, 2016 @04:54PM (#11136935)
If I run a car wash with a sign, "red cars washed free", and you paint your car red, have you defrauded me? You're not really a red car, you're just pretending!
Re:How Google’s Web Crawler Bypasses Paywalls by wmeredith on Friday February 19, 2016 @04:57PM (#11136950)
Oh pish posh. If you forge a key that fits the lock on my front door, yeah, your breaking the law. The lock itself is not an advertisement to come in if you can unlock it.
Re:How Google’s Web Crawler Bypasses Paywalls by fiatmoney on Friday February 19, 2016 @05:06PM (#11137011)
Stop it with the tired lock metaphors, they are wildly inapplicable. If the WSJ wanted to require access to some secret knowledge or possession of some token for access, they could. Instead, they are dispatching on something the protocol they're using agrees you are free to set to whatever you want for the "best experience". In fact the specification specifically recognizes that you will change the agent. Much like the color of your car.

"If a user agent masquerades as a different user agent, recipients can assume that the user intentionally desires to see responses tailored for that identified user agent, even if they might not work as well for the actual user agent being used"

It's a param exactly equivalent to asking "who would you like to be treated as?" If I say "the pope!" and you respond by kissing my ring, I have not defrauded you.

Re:How Google’s Web Crawler Bypasses Paywalls by TheDong on Friday February 19, 2016 @05:07PM (#11137020)
Terrible analogy.

You're talking about a sign with practically no legal meaning.

This discussion is about a law which does have legal meaning.

To fix your analogy, it would be "If there's a law that says only explicitly authorized cars are allowed in a car wash; else 5 years in prison, and I have a car wash and say 'only red cars' ...". Of course, that still doesn't properly capture this since the intent of the law also matters and that law would be senseless, and so interpretation would be less obvious.

The better analogy is the one below about breaking into homes being illegal, but what if you happen to have a key that fits the lock? (though that also is a bad analogy in its own way).

Basically, making physical analogies for technical matters is rarely correct. It's often the best way to convince non-technical people of a matter without them needing to actually understand it.

Re:How Google’s Web Crawler Bypasses Paywalls by fiatmoney on Friday February 19, 2016 @05:17PM (#11137093)
Fraud statutes do exist and the CFAA is in direct correspondence with them.

If there's anything the tech sector should be able to come to agreement on, it's that lock metaphors in a situation with absolutely no hidden knowledge or private tokens make us all dumber.

Re:How Google’s Web Crawler Bypasses Paywalls by pbhjpbhj on Saturday February 20, 2016 @09:31AM (#11140132)
The law seems to use trespass as the analogue for unauthorised access. We can argue it's a bad analogy, but it's the one we have.

The door is locked, the key is right there and all you have to do is pick it up and you can gain access, but that doesn't make you authorised to gain access.

The CFAA and UK CMA may well be overzealous in restricting what should (to technologists) be allowed, but it's what we have as statute in the respective jurisdictions.

Re:How Google’s Web Crawler Bypasses Paywalls by hughes on Friday February 19, 2016 @04:39PM (#11136834)
Is that significantly different from viewing google's cached result of their legitimate access? The result is the same.
Re:How Google’s Web Crawler Bypasses Paywalls by TheDong on Friday February 19, 2016 @05:09PM (#11137042)
The difference is intent!

The law says "authorized access" and the WSJ authorized Google to access their content in order to index it. The WSJ did not authorize that content to be presented, for free, to you, an end user, necessarily. I don't know which way a court would rule on it, but it's definitely not black-and-white.

Sure, it's technically similar, but the court doesn't care. The court doesn't care if the law makes no technical sense, because it's a law, not a program.

Re:How Google’s Web Crawler Bypasses Paywalls by chavesn on Friday February 19, 2016 @09:01PM (#11138277)
I agree with you that this is what the law seems to think (being overly broad), but not if you are arguing this is sensical.

In other words, yes, the WSJ does intend to only give Google access to their content, and not the general public.

But no, the WSJ has not "authorized" Google by anything more official than a bank telling their security guards to let anyone into the vault who is wearing a blue t-shirt.

So yeah, I agree with you, there is a lot of conflation of technical means and the law, but we also shouldn't be granting to the WSJ that they are doing any real "authorizing" here, beyond wishing it and hoping it stays true.