News for hackers. Stuff that matters.
Re:Verizon's accidental mea culpa by amalag on Thursday July 17, 2014 @02:08PM (#8049149)
I have to agree with this assessment. If Verizon is claiming they have green in their network with 34% utilization to Level 3, then the connection between Level 3 and Verizon is lacking.

Is it a simple fix like Level 3 & Netflix claim? If so, what is Verizon's advantage to continue the slow Netflix. Netflix is really slow for Verizon users, it is very obvious for all their subscribers.

Re:Verizon's accidental mea culpa by msandford on Thursday July 17, 2014 @02:33PM (#8049298)
Yes, it absolutely is a simple fix.

This is the increasingly common case of an ISP wanting to have it's cake and eat it too. It's an analogy of course, but it's a good one.

Verizon has either a monopoly or a near monopoly on the last mile from POPs to it's customers. In other words customers have no choice or little choice but Verizon.

And these customers pay Verizon for some amount of bandwidth and expect Verizon to make a good faith effort to deliver that bandwidth. The Terms & Conditions always say "up to X Mbps" because Verizon can't control network congestion once the data is off of Verizon's network. It might be that once the data leaves their network it has to go to Australia which has notoriously poor internet as a result of being kind-of out of the way. Or maybe it heads to Africa and starts making cell tower hops in which case it's going to be abysmal.

But in this case Verizon isn't making a good-faith effort to deliver the bandwidth that their customers are paying for to them. What they want is to extract extra money from someone who also provides a service to their customers, namely Netflix or Level3. Why do they want to do this? It's hard to say for sure. Trying to make extra profits, trying to keep Netflix slow and encourage people to subscribe to Verizon cable TV, spite? We can't know.

But what is clear is that this isn't the fault of Netflix or Level3. Level3 has said they will upgrade their side of the connection for free which would reduce congestion and Netflix has offered to locate servers on the Verizon network for free, which would reduce the Verizon/Level3 network congestion.

But Verizon refuses to accept either of those offers. Why? I'll venture a guess: it's not because they have the best interests of their customers in mind. If they did, they would have upgraded already.

To me it seems like they're abusing monopoly(ish) power.

Re:Verizon's accidental mea culpa by Zikes on Thursday July 17, 2014 @02:39PM (#8049335)
> Level3 has said they will upgrade their side of the connection for free which would reduce congestion and Netflix has offered to locate servers on the Verizon network for free, which would reduce the Verizon/Level3 network congestion.

They're not necessarily doing this "for free", but as a part of the service their customers have already bought and paid for. It's not doing anything extra, it's normal operations, it's keeping the promise the business made to the customer, which is the opposite of what Verizon is doing in this case.

Re:Verizon's accidental mea culpa by msandford on Thursday July 17, 2014 @02:41PM (#8049346)
Yeah that's a good point. What I meant was "without any charge to Verizon"
Re:Verizon's accidental mea culpa by Zikes on Thursday July 17, 2014 @02:54PM (#8049432)
Absolutely, and you made the point well. I think when companies start butting heads like this, some people don't recognize that it's not just some corporate bank account, but the results of the dollars they've already given for the services they've signed up for. I think that disconnect is responsible for a lot of the lack of outrage around this.
Re:Verizon's accidental mea culpa by digikata on Thursday July 17, 2014 @08:06PM (#8050963)
I wonder if Verizon's customers could form a class action lawsuit accusing it of failing in good faith to deliver bandwidth from Netflix due to intentionally congested handover points.
Re:Verizon's accidental mea culpa by snoopdug on Friday July 18, 2014 @03:57PM (#8055258)
http://media.mofo.com/files/Uploads/Images/130725-Employment...

"Thus, class-action waivers must be enforced regardless of the consequences for the vindication of small-value claims, state or federal."

also, http://www.cbs46.com/story/24388303/consumer-advocates-fine-...

also http://publicjustice.net/content/tribe-scotus-denying-access...

also http://www.minnpost.com/community-voices/2014/05/forced-arbi...

So any disputes with verizon cannot go to court, they have to be arbitrated. Further, there cannot be a class arbitration. So to vindicate your rights against verizon, you'd have to be willing to bring an individual arbitration against Verizon and bear the costs of doing so which will be exponentially more than the value of your claim. In essence, these clauses immunize corporations from liability under antitrust laws, consumer protection laws; employment laws etc. It is a terrible situation created by a Supreme Court that despises class actions.

Re:Verizon's accidental mea culpa by snoopdug on Friday July 18, 2014 @08:38AM (#8052661)
Verizon has been given immunity from the threat of a class action by virtue of its arbitration clause class action waiver. Same is true comcast, AT&T etc. You can thank the Supreme Court for this.
Re:Verizon's accidental mea culpa by PeterisP on Saturday July 19, 2014 @12:48PM (#8057969)
Aren't such waivers automatically null and void at least in some states? In EU, such disclaimers are simply a waste of ink, but I recall seeing that some USA states had gone that way as well.
Re:Verizon's accidental mea culpa by digikata on Friday July 18, 2014 @02:06PM (#8054604)
Hmm, I've heard of immunity over class action wiretap related issues, but bandwidth delivery too? Any references? "class action immunity verizon" just has too much noise.
Re:Verizon's accidental mea culpa by mikeash on Thursday July 17, 2014 @02:11PM (#8049166)
Verizon's advantage should be obvious: they also sell TV service which is in direct competition to Netflix. If Netflix works poorly, that's more customers potentially signing up for FiOS TV.
Re:Verizon's accidental mea culpa by lbotos on Thursday July 17, 2014 @02:24PM (#8049237)
I feel like this is what people often forget in the net neutrality debate. The pipe providers have a horse in the race (FiOS TV On Demand, XFinity On Demand) and we need to make sure they aren't rigging the race so to speak.
Re:Verizon's accidental mea culpa by mikeash on Thursday July 17, 2014 @02:33PM (#8049297)
Going beyond that, they typically have a monopoly or duopoly as an ISP, while they're subject to competition as a TV provider.

Much of net neutrality can be reduced to "don't let monopoly ISPs leverage that into a monopoly in other services".

If Verizon didn't offer TV, this wouldn't really be a problem. If there were 17 other broadband ISPs to choose from, this wouldn't really be a problem. But Verizon does offer TV and they're one of two choices for me and most of their other customers, so it's a problem.

Re:Verizon's accidental mea culpa by hga on Thursday July 17, 2014 @03:04PM (#8049496)
It's even stated parenthetically in the blog posting when comparing European and US Level 3 and Verizon peering utilization, the former ~18%, and the latter ~100% "(where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses)"
Re:Verizon's accidental mea culpa by Crito on Thursday July 17, 2014 @02:34PM (#8049304)
Furthermore, if Verizon degrades connections to whichever ISPs Netflix uses, then they have leverage to get Netflix to pay Verizon to peer. If they make it so that the only way that Netflix can offer their service is to pay them for it directly, then that is more money for Verizon. Very clear motivation.
Re:Verizon's accidental mea culpa by fixermark on Thursday July 17, 2014 @03:24PM (#8049630)
I'm not a lawyer, but when you put it that way, it almost screams "RICO violation," doesn't it? ;)
Re:Verizon's accidental mea culpa by higherpurpose on Thursday July 17, 2014 @02:31PM (#8049283)
They just want the same deal as Comcast, that Netflix, in a moment of stupidity, made:

http://knowmore.washingtonpost.com/2014/04/25/this-hilarious...

Re:Verizon's accidental mea culpa by xxchan on Thursday July 17, 2014 @03:10PM (#8049528)
It's not stupid at all. By giving into the racket, Netflix demonstrated that it's not a technical issue at all, so that argument no longer flies. It's clear that the ISPs are capable of supporting the service, they just don't want to.
Re:Verizon's accidental mea culpa by higherpurpose on Saturday July 19, 2014 @08:16AM (#8057292)
Well not really. They can just claim that without being paid extra to do so, it wouldn't be cost-effective for them or whatever. I mean that's their public argument anyway - that these services cost them too much and they need to pay up.
Re:Verizon's accidental mea culpa by yxhuvud on Thursday July 17, 2014 @03:50PM (#8049788)
This is not and has never been a technical problem. It is purely in the business and politics areas.

One way to see the conflict is that it is about what status verizon have - are they considered an end user isp or is it a transit provider? L3 is in the right if they should be considered the former, and Verizon is right if they should be the latter.

Re:Verizon's accidental mea culpa by shimshim on Thursday July 17, 2014 @02:09PM (#8049154)
Its not always great on Comcast either. It has gotten better over the last 6 months though, I will admit.
Re:Verizon's accidental mea culpa by spuiszis on Thursday July 17, 2014 @02:31PM (#8049279)
I have Comcast as well and it's gotten because of the deal Netflix struck with them... http://knowmore.washingtonpost.com/2014/04/25/this-hilarious...

EDIT: John Oliver also talks quite eloquently about this as well: https://www.youtube.com/watch?v=fpbOEoRrHyU

Re:Verizon's accidental mea culpa by amalag on Thursday July 17, 2014 @02:10PM (#8049164)
I bought a VPN and it fixes it. Sad that it is necessary.
Re:Verizon's accidental mea culpa by IgorPartola on Thursday July 17, 2014 @02:44PM (#8049367)
I currently have a 6in4 IPv6 tunnel through Hurricane Electric. At least netflix.com seems to load over IPv6. I wonder if my Netflix experience is fine precisely because my traffic appears to go to HE first, then to Netflix.

Edit: in the past I have had similarly good experiences with YouTube problems. They seem to go away as soon as you start accessing YouTube over IPv6.

Re:Verizon's accidental mea culpa by boredinballard on Thursday July 17, 2014 @03:01PM (#8049472)
Interesting. I'm going to have to start trying Netflix and Youtube over IPv6.
Re:Verizon's accidental mea culpa by ChuckMcM on Thursday July 17, 2014 @02:28PM (#8049261)
I did this on Comcast as well, but its interesting to note they do shoot downs of ssh tunnels. Using IPSEC hasn't been an issue though.
Re:Verizon's accidental mea culpa by Zikes on Thursday July 17, 2014 @02:41PM (#8049347)
Can you describe that ssh tunnel issue? Are they monitoring your traffic and actively disrupting anything that looks like ssh?
Re:Verizon's accidental mea culpa by ChuckMcM on Friday July 18, 2014 @01:55AM (#8051869)
Sure, set up an SSH tunnel, make it a SOCKS5 proxy (easy enough to do) to a host in the 'cloud', now proxy you traffic through it (like Netflix).

Watch how suddenly you get 'connection reset by peer', look around and discover nobody in your packet path has any cause to reset your traffic. Now do a straight http proxy (varnish works well for this) to the same machine, run it all day night with bits of http traffic. No issues at all.

No go back to creating an SSH tunnel between your machine and the endpoint. Run traffic, note the mysterious 'connection reset by peer' when some thing upstream sends your TCP connection a FIN.

It could be that their network just has a really hard time with encrypted packets but some how I don't think so. I have tried a variety of port numbers.

EDIT: and I went back and verified I still have 'ServerAliveInterval 60' in my config file for all hosts.

Re:Verizon's accidental mea culpa by ohashi on Thursday July 17, 2014 @02:46PM (#8049381)
When I was on Comcast, I felt like they were doing this to gaming traffic as well. I have no evidence to back it up except anecdotal experience though.
Re:Verizon's accidental mea culpa by Zikes on Thursday July 17, 2014 @04:50PM (#8050117)
When I was on AT&T U-Verse they did this with torrents. I replaced my router, OS, and modem thinking the problem lay somewhere on my end, and once I ruled all of that out I called up AT&T to complain and call out the fact that it happened only when I torrented.

Their response was that yes, they do disrupt all torrenting traffic, "because it is illegal and they must comply with copyright holders". Nevermind it was various linux distros I was trying to download and evaluate, all torrenting is automatically evil by association.

I had to switch to Cox because they're the only other option in my area. The 250GB data cap on my plan is BS, but at least I can spend that 250GB however I please.

Re:Verizon's accidental mea culpa by digikata on Thursday July 17, 2014 @08:33PM (#8051071)
I had problems a while back with any long-running downloads or streaming video on ATT DSL. Basically made some uses impossible - any large remote data that couldn't resume at arbitrary points. Went completely away when I switched to a third-party ISP - using the same ATT-owned DSL lines.
Re:Verizon's accidental mea culpa by Zikes on Thursday July 17, 2014 @08:51PM (#8051130)
I signed up for a VPN service right at the end, and that cleared everything up for me and completely confirmed they were intentionally shaping and disrupting traffic.
Re:Verizon's accidental mea culpa by silverbax88 on Thursday July 17, 2014 @02:26PM (#8049251)
Which VPN did you go with?
Re:Verizon's accidental mea culpa by amalag on Thursday July 17, 2014 @02:42PM (#8049362)
I used privateinternetaccess, i am happy with it. I don't even use it all the time but the speed tests hold up against my 50Mb/s connection. Hulu has blocked it if that is a concern, there is some other VPN that has more egress IP's that works with Hulu, but I didn't know about it and it was more expensive anyway.
Re:Verizon's accidental mea culpa by djchen on Friday July 18, 2014 @01:20AM (#8051790)
You can run your own VPN on a VPS and have a dedicated IP without needing to worry about Hulu getting blocked.
Re:Verizon's accidental mea culpa by induscreep on Thursday July 17, 2014 @02:15PM (#8049188)
Verizon probably needs to pay Level 3 for the bandwidth - so apart from the cost of the hardware (cards + cables) there is a possibly large cost which is not mentioned in the Level 3 article.
Re:Verizon's accidental mea culpa by ktsmith on Thursday July 17, 2014 @02:24PM (#8049236)
Verizon probably doesn't need to pay Level 3 for the bandwidth. Typically peering points like this are settlement free. In other words it's a voluntary exchange of traffic and each side of the line keeps 100% of the revenue from its customers using the interchange.
Re:Verizon's accidental mea culpa by mikecb on Thursday July 17, 2014 @02:37PM (#8049329)
And Verizon claims that peering points are intended to be roughly equal in directional usage. Which this is not. Are there examples of peering agreements where companies don't care that usage isn't balanced and still don't require some sort of net settlement?
Re:Verizon's accidental mea culpa by wmf on Thursday July 17, 2014 @02:41PM (#8049340)
Peering links between backbones (like Level 3 and Verizon Business) are supposed to have equal ratios. Peering between content providers and end-user ISPs don't care about ratios.
Re:Verizon's accidental mea culpa by tromobne8vb on Thursday July 17, 2014 @03:16PM (#8049577)
Right. If they really cared about ratios they wouldn't offer unbalanced service such as 50/15. They have built their network around users predominately consuming content and then complain when there is too much consumption from specific providers that happen to compete with services they also offer.
Re:Verizon's accidental mea culpa by mikecb on Thursday July 17, 2014 @02:46PM (#8049382)
Right. And the latter has to pay for it.
Re:Verizon's accidental mea culpa by wmf on Thursday July 17, 2014 @02:49PM (#8049401)
The latter what? I think content peering should be free because there's no fair way to set a price for it.
Re:Verizon's accidental mea culpa by mikecb on Thursday July 17, 2014 @02:59PM (#8049468)
Why not? There's plenty of price data for high volume bandwidth applications, and even if there weren't, the parties could negotiate for it.
Re:Verizon's accidental mea culpa by wmf on Thursday July 17, 2014 @03:07PM (#8049510)
An end-user ISP like Comcast has a monopoly on access to their customers, so if they're allowed to charge for peering there is no competition to limit how much they charge. This tends to result in the ISP allocating 99% of the surplus created by peering to themselves.
Re:Verizon's accidental mea culpa by mikecb on Thursday July 17, 2014 @03:26PM (#8049651)
That's very true. The only limit would be losing customer (HA!) because they didn't connect to the right content providers. There's some interesting network effects here that smell like positive externalities, which would explain the underinvestment. But also suggest that there isn't a good solution.
Re:Verizon's accidental mea culpa by ktsmith on Thursday July 17, 2014 @02:50PM (#8049412)
I'd say that wmf addressed it already. Peering points with one side being primarily consumer services are always going to be unequal due to the nature of the end user and the ways in which consumer ISPs provide bandwidth to their customers. Lots of download, and very little upload speed as an example. As such it's nearly impossible for many peering points primarily used by consumer services to use an equal amount of traffic on both directions. Both sides of the agreement know this and it's still usually in both of their best interests to peer settlement free.
Re:Verizon's accidental mea culpa by mschuster91 on Thursday July 17, 2014 @02:41PM (#8049345)
I'd say a lot of the peerings at DECIX and AMS-IX.
Re:Verizon's accidental mea culpa by mikecb on Thursday July 17, 2014 @02:50PM (#8049413)
Good to know. And these all remain settlement-free?
Re:Verizon's accidental mea culpa by mschuster91 on Thursday July 17, 2014 @04:10PM (#8049896)
iirc it is part of the peering culture there, but I am not sure exactly, never having worked in that field
Re:Verizon's accidental mea culpa by bitJericho on Thursday July 17, 2014 @02:18PM (#8049207)
Netflix has offered to directly connect to Verizon without the need for Level3 to be in the middle.
Re:Verizon's accidental mea culpa by btgeekboy on Friday July 18, 2014 @01:52AM (#8051863)
This is a good point, but for a different reason: IIRC, Netflix is paying Level3 for connectivity. If Netflix pays Verizon for direct connectivity, Netflix still pays someone for bandwidth, and L3 loses their cut.
Re:Verizon's accidental mea culpa by spyder on Thursday July 17, 2014 @03:09PM (#8049523)
and Verizon wrote in their post that they "working" on this: "we are working aggressively with Netflix to establish new, direct connections from Netflix to Verizon’s network." But "working" could mean: "waiting for Netflix to pay for it".
Re:Verizon's accidental mea culpa by wmf on Thursday July 17, 2014 @03:32PM (#8049685)
Netflix already paid Verizon's ransom and now Verizon is building connects to Netflix but such connections take time to install. Level 3 is arguing over old news at this point, but they're trying to set a precedent to prevent other ISPs from trying the same thing.