News for hackers. Stuff that matters.
Re:Level3 is without peer, now what to do? by mhandley on Tuesday May 20, 2014 @12:43PM (#7773547)
Would be very interesting to see what happened if the big CDN providers just depeered Comcast for 24 hours. Would certainly cost Comcast a fair amount in customer support calls, bad publicity, and properly bring the debate to the general public.
Re:Level3 is without peer, now what to do? by eldavido on Tuesday May 20, 2014 @03:04PM (#7774514)
This would be an unmitigated disaster. Everyone would call Comcast, Comcast would tell the story their way, and things would be even worse than they are now. They'd probably get on national nightly news talking about how their "partners" "unfairly" "cut them off from the Internet".

It would be pretty interesting to watch the spin machine in action, though. 100% pure PR spin.

Re:Level3 is without peer, now what to do? by ryanhuff on Tuesday May 20, 2014 @04:08PM (#7774865)
That's why you throttle Comcast customers. Can't it go both ways?
Re:Level3 is without peer, now what to do? by sirclueless on Tuesday May 20, 2014 @10:53PM (#7776464)
The point is that if Netflix throttles Comcast customers, the customers stream from Comcast On-Demand instead. They don't switch away from Comcast because they can't, which is what makes this an unbalanced situation.
Re:Level3 is without peer, now what to do? by jarjoura on Wednesday May 21, 2014 @12:28PM (#7779108)
Also, don't forget that Comcast now owns NBC. :-)
[deleted]
Re:Level3 is without peer, now what to do? by stonemetal on Tuesday May 20, 2014 @01:22PM (#7773803)
content delivery networks (Hulu, YouTube, Netflix, etc) , on the other hand, are constantly at risk of being replaced.

L3 is a CDN, Hulu is not. Hulu hires a CDN to get their data out there. CDNs are basically ISPs that provide high bandwidth with low latency by providing a distributed cache. Netflix has gotten in to the CDN business because at their scale none of the CDNs have been able to keep up with demand, or were not willing to pay for peering with last mile ISPs.

Re:Level3 is without peer, now what to do? by zimbatm on Tuesday May 20, 2014 @02:14PM (#7774183)
The ones who have the eyeballs are the service sites like Netflix, Hulu and Youtube. If they wanted they could influence public opinion quite easily by showing a message educating the user why their service is slow. "Your ISP has decided to keep 60% of the money you gave to them instead of upgrading their infrastructure. Now they want us to pay their infrastructure upgrade. We wish that our service was faster but don't want to get bullied into it. Please help us by contacting your ISP's customer support."
Re:Level3 is without peer, now what to do? by mhandley on Tuesday May 20, 2014 @01:27PM (#7773851)
Monopolies are not so difficult to break. You just need to convince the politicians that doing so will get them reelected. And to do that you need to get the public really annoyed. Now how could you possibly do that I wonder <grin>
Re:Level3 is without peer, now what to do? by hga on Tuesday May 20, 2014 @01:46PM (#7773989)
More accurately, you convince them not breaking or adequately regulating the monopoly will force them to spend more time with their families after the next election. In the experience of gun owners, this works rather well after you've actually done enough times, e.g. every bi-annual national election from 1994-2000.
Re:Level3 is without peer, now what to do? by dfc on Tuesday May 20, 2014 @01:38PM (#7773925)

  > This is asymetrical warfare.
When I got to that sentence I thought "Really, a war comparison?". I couldn't figure out where you were going to go with the comparison but I thought I would play along. After reading your comment a couple of times I still can't figure out what the connection is to asymmetric warfare. Who are the belligerents and the casus belli? How is this different from traditional "telecom warfare"?

This is one of the more bizarre reading experiences on HN:

I don't think that is the solution. This is asymetrical warfare...The solution to this will be worked out by the free market slowly but surely.

Re:Level3 is without peer, now what to do? by freehunter on Tuesday May 20, 2014 @02:31PM (#7774285)
Really? It's an off-the-cuff remark to describe a fight that's slowly playing out. It's a pretty common phrase.
Re:Level3 is without peer, now what to do? by dfc on Tuesday May 20, 2014 @03:01PM (#7774490)
Yes, really. I have never seen the free market be offered as "the solution" to a metaphoric asymmetric conflict. It seems like there is an inherent existential crisis that has prevented people (up until now) from suggesting the free market as a solution to asymmetric warfare. Is it really a free market when such large asymmetries exist between market participants?

That being said I find the suggestion that the colloquial definition of "asymmetric warfare" is "a fight that's slowly playing out" to be even stranger. Do you think that is the definition of "asymmetric warfare"? I must be reading different works than you because I cannot remember seeing "asymmetric warfare" used--and most certainly not commonly used--as a placeholder for a fight that is slowly playing out.

Re:Level3 is without peer, now what to do? by freehunter on Tuesday May 20, 2014 @04:49PM (#7775091)
First you were arguing that this should not be called warfare. Now it seems you're arguing something completely different. I honestly have no idea what you're talking about, sorry.
Re:Level3 is without peer, now what to do? by dfc on Tuesday May 20, 2014 @05:14PM (#7775231)
I never argued that this should not be called warfare. I said that I did not think that this was a good example of asymmetric warfare. To which you responded that "asymmetric warfare" was commonly used to describe a "fight that is slowly playing out." This was also confusing to me. I will restate my queries in order to clear up any confusion:

  1. How is the CDN/ISP conflict an example of "asymmetric warfare"?
  2. Do you think the definition of "asymmetric warfare" is "a fight
     that is slowly playing out"?
Re:Level3 is without peer, now what to do? by Pwntastic on Tuesday May 20, 2014 @01:04PM (#7773696)
Perhaps, but something has to give. Maybe directly pissing off 4 million customers is what's needed to get these monopolies broken?
Re:Level3 is without peer, now what to do? by danielweber on Tuesday May 20, 2014 @01:31PM (#7773892)
Or people might wonder what gives these CDNs the right to declare warfare against their ISP and demand regulation of the CDNs.
Re:Level3 is without peer, now what to do? by aroch on Tuesday May 20, 2014 @01:05PM (#7773706)
> The solution to this will be worked out by the free market slowly but surely. The CDNs will eventually wise up - like Google - and get into the ISP business.

CDN are in the ISP business.

Re:Level3 is without peer, now what to do? by shiftpgdn on Tuesday May 20, 2014 @02:04PM (#7774113)
At this point this seems to be the ONLY solution. Knocking Comcast off the internet for 24 hours would result in a lot of attention towards net neutrality and their lackadaisical attitude towards peering.
Re:Level3 is without peer, now what to do? by diminoten on Tuesday May 20, 2014 @02:27PM (#7774263)
They wouldn't be knocked off the Internet, they'd just have to route their packets through less efficient paths, which would slow everything down significantly.

The Internet would more or less shut down from traffic overload if all of Comcast's customers lost all the shortcuts that Comcast has created for them to content.

Re:Level3 is without peer, now what to do? by jeremyjh on Tuesday May 20, 2014 @02:49PM (#7774409)
Well the CDN could alternatively blackhole these networks.
Re:Level3 is without peer, now what to do? by runeks on Tuesday May 20, 2014 @04:26PM (#7774956)
What's the purpose?

We're getting to the root cause of the problem here: lack of competition.

These ISPs already have the lowest customer ratings. If the customers could switch to another ISP they would.

This stunt would amount to kicking a man that's lying down. He's already used to the shitty quality of his Comcast connection, but now he's being punished by CDN providers without being able to do anything about it.

Re:Level3 is without peer, now what to do? by ilaksh on Tuesday May 20, 2014 @08:16PM (#7775993)
The reason there is no competition is because of regulations and physical constraints. We need ultra wideband mesh nets. Or some kind of single strand fiber with a glue that attaches it to the street and a new law that says that laying fiber strands is a natural right of every citizen.
Re:Level3 is without peer, now what to do? by edraferi on Tuesday May 20, 2014 @11:10PM (#7776515)
I very much think broadband internet should be treated as a basic utility. however, I feel that "natural right" takes it too far. it's a very high bar to reach
Re:Level3 is without peer, now what to do? by klodolph on Tuesday May 20, 2014 @12:54PM (#7773630)
It would also give the CDNs a fair amount in customer support calls, bad publicity, and make their paying customers angry about lost business.
Re:Level3 is without peer, now what to do? by ep103 on Tuesday May 20, 2014 @12:57PM (#7773652)
I don't know about bad publicity though. To each customer support call simply say, Comcast has broken their contract with our CDN. Please call comcast. Considering Comcast is one of the two most hated companies in America right now, by a _huge_ margin, people will easily believe that, especially since in this case, it would be true. Level3 et all are enabling this behavior.
Re:Level3 is without peer, now what to do? by chimeracoder on Tuesday May 20, 2014 @01:05PM (#7773705)
> Considering Comcast is one of the two most hated companies in America right now, by a _huge_ margin

And Time Warner is the second: http://bgr.com/2014/05/20/comcast-twc-customer-satisfaction-...

Nobody likes the cable companies. It would be very easy for someone to capitalize on that hatred to win public support for net neutrality; it's just a matter of crafting the right campaign.

Re:Level3 is without peer, now what to do? by coldpie on Tuesday May 20, 2014 @01:11PM (#7773738)
This is why I'm so disappointed to see the small-medium ISPs hop right in bed with Comcast and co-sign anti-net neutrality letters:

http://arstechnica.com/business/2014/05/top-isps-threaten-to...

With the national attention this issue is getting, this is a tremendous marketing opportunity to take customers away from the big dogs.

Re:Level3 is without peer, now what to do? by vanwesson on Tuesday May 20, 2014 @01:39PM (#7773935)
The smaller ISPs are just following Comcast's example: "You mean we can charge multiple times for the same data without doing any extra work? Sign us up!" All last-mile ISPs will eventually realize they can hold everyone hostage, so long as there is no significant competition.
Re:Level3 is without peer, now what to do? by aroch on Tuesday May 20, 2014 @02:01PM (#7774091)
Small ISPs also like this because they typically are very heavy on paid, private internet and don't do much peering due to having small, localized networks. They stand to gain, proportionally, the most since they can charge the people the were previously paying for access.
Re:Level3 is without peer, now what to do? by Spooky23 on Tuesday May 20, 2014 @01:17PM (#7773775)
Local Television stations and sports teams have successfully used this as a way to get cable companies to fold over contracts for carriage.

Everyone despises cable companies, most of the angst would go their way. After all, 9/10 people don't know what a CDN is!

Re:Level3 is without peer, now what to do? by rhizome on Tuesday May 20, 2014 @01:50PM (#7774012)
Support calls from whom? CDNs effectively sit behind the Comcast site.
Re:Level3 is without peer, now what to do? by noselasd on Tuesday May 20, 2014 @02:15PM (#7774192)
From the CDN customers. They'll be annoyed if they find out the service they paid money for isn't doing what it's supposed to do.

Since they've paid the CDN, they don't care what Comcast has done or is doing - they paid the CDN, it's the problem of CDN.

Re:Level3 is without peer, now what to do? by rhizome on Tuesday May 20, 2014 @02:27PM (#7774256)
Those customers aren't getting depeered, only Comcast.
Re:Level3 is without peer, now what to do? by tonyarkles on Tuesday May 20, 2014 @02:52PM (#7774436)
I'm not sure of a more eloquent way to say this. Comcast's customers are the CDN's customer's customers.
Re:Level3 is without peer, now what to do? by apetresc on Tuesday May 20, 2014 @03:07PM (#7774530)
To play devil's advocate, those CDN customers would also be unable to access content if their power went out. But nobody would think to say the CDN isn't doing their job if Texas had a blackout, because "I'm paying you to deliver my content to Texas, dammit!"

It's a bit of a stretch, but I think that's the analogy they're going for.

Re:Level3 is without peer, now what to do? by rhizome on Tuesday May 20, 2014 @03:33PM (#7774683)
Right, but that relationship, especially as applies to support and subscription, is not transitive.
Re:Level3 is without peer, now what to do? by stephengillie on Tuesday May 20, 2014 @10:34PM (#7776425)
How do you depeer my customers' provider without depeering my customers? How do you cut off a router's network access without cutting off access to the networks behind it? How do you blacklist a /8 but not one of its /24s without an exemption list? How do I prevent Bob from forking my code, but allow people who fork from Bob to fork the modifications he would do to my code? How do you join tables on 2 databases when you can't authenticate to the first?
Re:Level3 is without peer, now what to do? by stephengillie on Tuesday May 20, 2014 @10:29PM (#7776406)
I'd be FURIOUS if my CDN cut off service to part of my customer base, so they could attack one of my customers' mutual other vendors. And you better believe our ticket system would overload with angry tickets about how our web platform is broken again.
Re:Level3 is without peer, now what to do? by letstryagain on Tuesday May 20, 2014 @07:37PM (#7775853)
Not nearly as much as Comcast though. If I have my home internet with Comcast and I suddenly can't access Netflix while my friends on other ISPs can access it just fine, then of course I'm going to call Comcast.
Re:Level3 is without peer, now what to do? by specialp on Tuesday May 20, 2014 @06:06PM (#7775527)
Well then all of the customers of said transit providers would leave and switch to one that can reach the millions of Comcast customers. As Level 3 said if there is a problem with peering amongst transit providers someone can switch to another provider. If there is a problem reaching Comcast customers there is nothing the customer can do or the transit provider as there is only one route to that customer. That is the leverage Comcast has now.

If the end users had multiple options they would switch providers if connections were unbearably slow to YouTube or Netflix. But we know they do not so regulation is needed.

Re:Level3 is without peer, now what to do? by JTon on Tuesday May 20, 2014 @12:55PM (#7773635)
Too risky, no? If Comcast said the problem was not with their network, but with bit CDNs. Customers would shift their rage towards them instead. The whole thing would turn into a PR pissing match
Re:Level3 is without peer, now what to do? by jonknee on Tuesday May 20, 2014 @01:10PM (#7773734)
I'm going to go with customers being mad with who they send a big check to every month (e.g. Comcast). 99% of people only know about their ISP, not their ISP's providers.
Re:Level3 is without peer, now what to do? by Jtsummers on Tuesday May 20, 2014 @01:25PM (#7773825)
Not a big check, but they'd also be frustrated with Netflix/Hulu/Amazon Prime whom they also send money to each month (or year). The average customer wouldn't understand that it's the middleman causing problems, not one of the two endpoints (service provider or content provider).
Re:Level3 is without peer, now what to do? by jimbokun on Tuesday May 20, 2014 @02:14PM (#7774188)
If ABC has a blank channel, but you can still watch CBS and NBC and ESPN just fine, the average consumer will assume the problem is with ABC.

If all the channels are blank, the average consumer will blame the cable company.

Similarly, if "the Internet is down", from the customer's perspective, they're going to blame the ISP, not Netflix/Hulu/Amazon.

Re:Level3 is without peer, now what to do? by busterarm on Tuesday May 20, 2014 @05:28PM (#7775345)
Not true. I work for a cable/fiber ISP. Customers always blame the ISP.

Even when the problem is the feed (like with CW's networks last month)...or when some dipshit construction company digs where they aren't supposed to.

Luckily I don't work for one of the horrible ISPs that everyone hates, but we're definitely more in the blame-receiving business than is warranted.

Folks don't understand the complexity involved in making a utility service "just work" and if the provider doesn't have hundreds of millions to invest in infrastructure (like my small provider doesn't have) it won't.

The big ISPs have no excuse for not peering properly, but this is a fight they've been having with Level3 since the late 90s. Sadly they're winning.

Re:Level3 is without peer, now what to do? by singlow on Tuesday May 20, 2014 @08:48PM (#7776114)
When the customers call you, it is because they blame you. You never hear from the ones who don't blame you.
Re:Level3 is without peer, now what to do? by busterarm on Wednesday May 21, 2014 @10:50PM (#7781879)
That's not entirely true, but the majority of them do.
Re:Level3 is without peer, now what to do? by couchand on Wednesday May 21, 2014 @12:43AM (#7776729)
Wait, did you just call broadband internet a "utility"? As in a Title II utility?
Re:Level3 is without peer, now what to do? by busterarm on Wednesday May 21, 2014 @10:47PM (#7781873)
We provide phone service as well/primarily.
Re:Level3 is without peer, now what to do? by Jtsummers on Tuesday May 20, 2014 @02:27PM (#7774259)
That's if everyone particpates on a Comcast/TWC/Cox/etc. boycott. If my parents can get to Amazon Prime but not Netflix, they'll assume the issue is Netflix. (ok, maybe not my parents, but the idealized tech-unaware parent)
Re:Level3 is without peer, now what to do? by hardoncollider on Tuesday May 20, 2014 @03:07PM (#7774527)
But with such massive depeering as this, practically all of the internet would be affected, given the amount of traffic other peers would have to pick up.
Re:Level3 is without peer, now what to do? by jonknee on Tuesday May 20, 2014 @01:53PM (#7774038)
I guess that depends on the number of participating Tier-1 providers that pull the plug, if enough did it would be very obvious it wasn't Netflix with the issue because everything would grind to a crawl.
Re:Level3 is without peer, now what to do? by icebraining on Tuesday May 20, 2014 @01:59PM (#7774084)
What about the CDN's customers, ie., companies with sites that their costumers couldn't access?
Re:Level3 is without peer, now what to do? by Spooky23 on Tuesday May 20, 2014 @01:17PM (#7773785)
Ask your mom who her favorite CDN is.
Re:Level3 is without peer, now what to do? by Nexxxeh on Tuesday May 20, 2014 @08:49PM (#7776122)
Akamai. Moms love Akamai.
Re:Level3 is without peer, now what to do? by ryanhuff on Tuesday May 20, 2014 @04:09PM (#7774873)
Comcast's customers have no idea what a CDN is. All they know is that the internet is slow, and Comcast is the internet to them.
Re:Level3 is without peer, now what to do? by rhizome on Tuesday May 20, 2014 @01:51PM (#7774022)
I don't think it would go that way, their customers would more often say, "what's a CDN?"
Re:Level3 is without peer, now what to do? by JTon on Tuesday May 20, 2014 @02:28PM (#7774269)
They don't have to say CDN explicitly. They can simply say Netflix or Hulu. For example, "Dear valued customer. The recent slow down of your Netflix service is the result of an issue with Netflix servers. ISP_X service is fully functional. Please contact Netflix customer support for further assistance"
Re:Level3 is without peer, now what to do? by yebyen on Tuesday May 20, 2014 @04:20PM (#7774920)
I'm pretty sure Netflix just paid Comcast, and maybe I'm not keeping up with the issues as they arise and I'm really uninformed about the current state of things, but if Level3 is still having a dispute with Comcast, and Comcast blames Level3's actions on Netflix just because they need to blame someone who their customers will recognize...

Forget about the gigundous case for libel Netflix would have against Comcast on millions of counts... those Netflix payments that Comcast just earned are going to stop, and their first high-profile experiment with double-dipping payments will have failed abruptly.

I assure you Comcast does not want that.

Re:Level3 is without peer, now what to do? by pdonis on Tuesday May 20, 2014 @08:16PM (#7775991)
> I'm pretty sure Netflix just paid Comcast

They most certainly did; and what's more, the transit provider Netflix was using prior to paying Comcast was, IIRC, Cogent, not Level3.

Re:Level3 is without peer, now what to do? by yebyen on Wednesday May 21, 2014 @02:44PM (#7779977)
So, then this whole idea that Level3 is the only one having problems with these five last-mile ISPs is bunk? Level3's wikipedia page still says they deliver Netflix. Now you're saying they don't even host Netflix. What are Level3's other major clients? I have to imagine there are bigger fish than Apple/iTunes (maybe I'm all wet), they are the only other major client mentioned on the wiki page.
Re:Level3 is without peer, now what to do? by pdonis on Wednesday May 21, 2014 @06:08PM (#7781051)
> this whole idea that Level3 is the only one having problems with these five last-mile ISPs is bunk?

I think so, but it's only fair to note that it's very difficult to get accurate information about this, since all of the parties have a strong interest in not making all the details public, at least not while they're jockeying for position.

> Level3's wikipedia page still says they deliver Netflix.

No, it says (if you go look at the reference that's footnoted) that Netflix was about to switch to Level3 (from Cogent, although that specific reference doesn't say so) as as transit provider, and Comcast used that to try to extract peering fees from Level3 based on an expected increased level of traffic. When Level3 refused, Comcast went to Netflix and got them to pay for direct connection to Comcast's network, bypassing any transit provider.

Re:Level3 is without peer, now what to do? by rhizome on Tuesday May 20, 2014 @08:38PM (#7776077)
Um, that would be a trade libel.
Re:Level3 is without peer, now what to do? by larrys on Tuesday May 20, 2014 @01:59PM (#7774081)
Can't do that. These are corporations with vast paper trails.

The concept of "Tortious interference" [1] comes to mind but I'm certain there are other legal theories that could be employed by Comcast.

[1] http://en.wikipedia.org/wiki/Tortious_interference

Re:Level3 is without peer, now what to do? by rosser on Tuesday May 20, 2014 @02:52PM (#7774435)
I think the point is that Comcast is already engaging in tortious interference, and then trying to charge the upstream provider for the privilege. Sure, they could file suit on those grounds, but the countersuit would follow very quickly, and it would be joined by everyone.
Re:Level3 is without peer, now what to do? by JoshTriplett on Tuesday May 20, 2014 @05:28PM (#7775341)
> Would be very interesting to see what happened if the big CDN providers just depeered Comcast for 24 hours.

Those providers would very quickly get prosecuted via collusion and antitrust laws, and Comcast would gain a moral high ground to stand on, almost certainly making the situation worse.