* This would increase throughput and capacity for both Level3 and the consumer networks.
2) Netflix has offered free Open Connect Appliances to all major ISP's.[2]
* This would allow consumer networks to only download a video once, then serve it up locally within their network to all their customers. In-network traffic is almost free for the ISP's.
3) Netflix has offered to change their entire distribution model into a P2P model.[3]
* This would allow consumer networks to only download a video once, then serve it up locally within their network to all customers. In-network traffic is almost free for ISP's.
.
All of these options have been flat-out refused. The Big5 ISP's are purely after the money. There is no other compromise for them.
[1] http://blog.level3.com/global-connectivity/chicken-game-play...
[2] https://www.netflix.com/openconnect/hardware
[3] http://arstechnica.com/information-technology/2014/04/netfli...
I work at an ISP, can confirm. All decisions about capacity are about how much money we'll make or lose. We're intentionally keeping bandwidth constrained because we know it will severely impact our cable TV business.
Net neutrality simply means they are not allowed to go into homes and figure out what you're using your electricity for, and try to charge you more based on that usage.
It gives them pops, "free" capacity (meaning they have to buy dark fiber, and can resell internally cheaply, or just wouldn't have it at all if they didn't have phone business), shared infrastructure.
So if you do this, effectively it will mean the same any other solution means : more expensive internet.
Given that that will be the result, might I suggest an alternative solution ? Get a basic business internet line from Verizon. See the difference.
like what exactly ?
All of Netflix, in HD, on 24TB.
Netflix could solve the bandwidth problem by just mailing the whole library to customers. "Never underestimate the bandwidth of a station wagon full of magnetic tapes."
They will put a server in a data center anywhere an ISP wants, and they'll manage the whole syncing thing so the ISP doesn't even know it's there. After that, all content comes from that cache.
My personal opinion is that you should cache/co-locate your data close the the edge of the edge of your network whenever it presents a benefit for congestion and price.
http://www.listbox.com/member/archive/247/2014/07/search/bmV...
Its the senders responsibility to deliver data to the local ISP of the recipient. It doesn't matter if the person asked for the data, you still have to pay to get it there.
The issue is that Verizon is both a national transit provider and a local ISP. If you just dump your data on the nearest Verizon peering port, you aren't fulfilling your end of the bargain. You are asking Verizon to do it for you, as a transit provider.
However, if you are delivering it to the last mile, you have "paid for it" it already.
The issue is where are these congestion issues occurring. At the last mile or at the edges of Verizon's transit network.
Verizon absolutely has the right to bill for providing transit.
Even if we agree that Verizon has the right to bill Level 3 for transit within _Verizon's own network_, it seems odd that they are not taking advantage of Netflix's CDN boxes, which would drastically reduce the amount of transit required.
If you just dump your data on the nearest Verizon peering port, you aren't fulfilling your end of the bargain.
So, let's say I build a new skyscraper with 10,000 apartments, and I set myself up as the exclusive ISP to the tenants of that skyscraper. I wire up the building nicely, and have some big switches and routers in the basement. Does every content provider (e.g. Netflix) or company working on their behalf (e.g. Level 3) now have an obligation to build new circuits, and deliver the traffic all the way to my basement?
For years, this is how Verizon/AT&T/SBC/BellSouth/etc had structured telephone networks. And this is probably the same structure they want to extend to shared data networks aka the Internet.
If you were a competitive telco carrier who wanted to deliver a massive amount of phone calls to Verizon customers in a particular region, you couldn't just dump it off at the tandem (which you can think of like a telecom peering point), you had to extend your network to the end offices where the Verizon customers were aggregated.
The thought there being there's finite capacity between the end office and the tandem. And if you're going to use most of it, you should either pay for it or build your own facilities to alleviate congestion.
You're dealing with a traditional telecom company (Verizon) in its telecom mindset (build to me/mid-span meet, and keep a very tight watch on ratios/meet points via accounting, billing, and state regulation), vs. a traditional Internet company (Level 3) in its traditional peering mindset (build to the exchange, then build to me, and keep a loose observation on ratios/meet points with no regulation).
I hope I haven't gone too far off track.
The state won't let you take on calls as a local access provider if they're going to end up going nowhere.
You also have a regulated obligation to provide working 911 service so you need to interconnect with the incumbent's 911 tandem switches. So, off the bat, you have multiple connections with or to Verizon.
Now, if there are some really popular people in your building who attract a certain amount of calls, YES, you can demand that the other telecom carriers extend facilities to your switch (which is located in your basement).
Those carriers will decide if they want to build facilities to you, or work with another carrier (remember the one interconnection agreement you had to sign?), to pay them to deliver calls to you on the agreement that they have enough facilities with you.
You are compensated, per minute for any call that terminates on your skyscraper network through a process called reciprocal compensation, whose rates are laid out in the interconnection agreement you signed.
Is old-timey Verizon's attitude starting to make sense?
How quickly would this change Verizon's policy?
As a customer of an ISP, I pay for access to servers on the internet. My ISP charges me for a level of bandwidth. If they are the bottleneck slowing down my connection to the services I want to access, they should fix it, and give me what I'm paying them for.
For example, you want to download something off a Server in Germany. Your ISP is responsible for getting your initial message to the server. But it's not responsible for taking that information in Germany and carrying to back to the US for you. The Server's ISP has to carry it to the USA and then dump it off at your local ISP.
Now your ISP and the Server ISP may have a peering point in Germany, but they might not. But either way, your ISP isn't under any obligation to pay for transit of data being sent to you. That's just not how it works. And that definitely isn't what is in your agreement with your ISP.
Is that accurate? Or no?
In this case Netflix is being forced to pay for transit from their "house" all the way to mine. Not how the internet normally works.
AWS may be the origin for content distributed via peering, appliances, and CDN partners but your videos aren't streaming from AWS: https://www.netflix.com/openconnect
you can't put your own hardware at AWS.
they are the poster child for using AWS for what it is good for, general purpose compute (storing your play list, the list of all movies, controlling the CDN etc)
but I imagine for streaming you need bare metal
The agreement between Level3 and Verizon probably has some balanced traffic requirement, with penalties for asymmetry. If that is the case, then the nature of the imbalance would force Level3 to make payments to Verizon.
Verizon gave a very clear suggestion for Netflix: work with other providers as well. If Netflix worked with the overwhelming majority of third party providers, then you could argue that Verizon is not playing fair. But I don't see any commentary that Netflix is actually working with most of the third party providers.
So does that mean there's a downloader-pays rule?
If so that means that Verizon should be paying Level3 because it consumes 3x the bandwidth that it provides.
Verizon's needs Level3 3x as badly as Level3 needs Verizon on a bit-by-bit basis. So why isn't Verizon paying Level3 for the imbalance?
There are actually two sides to the ISP game. There's the consumer ISP side and there's the commercial ISP side.
Consumer ISPs are in business to get their customers hooked up to as much of the internet as possible to enable them to consume things. And they charge their customers (normal folk like us) for this service.
Commercial ISPs are in business to get their customers hooked up to as much of the internet as possible to provide services. And they charge their customers (big companies) for it.
When commerial ISPs meet consumer ISPs typically they say "let's trade traffic for free so that way we can both keep our money and not worry about accounting"
What's happened is that a consumer ISP is trying to charge more than just it's last-mile normal customers for data that they use and which it is ostensibly contractually obliged to provide to them without further fees.
What Verizon is saying is "look they need to pay us to upgrade this link" which on the surface doesn't sound too bad. It kinda makes sense. If I used more bandwidth they would charge me more. But the thing is that I am already paying them to make sure that the connection to the internet is uncongested. They are deliberately allowing it to get congested to try and bill both sides of the transit.
They may have a peer balancing agreement, which is not being held up, but they're also still being payed by their customers to provide access to content outside of their own network and Verizon is enforcing a limitation from another agreement (the peering agreement) that is affecting their ability to provide the content their customers want to have access to.
You can't have it both ways. Or... I guess you can, as Verizon has shown.
This obviously gives the Consumer ISPs pricing power in all of their business dealings (including any peering contracts that they force the Commercial ISPs to sign). The fact that Verizon owns RedBox (a Netflix competitor) & sells TV packages is just further motivation for hampering the interconnection.
On the commercial side they provision a link from point A to point B (which is what his software figures out) and it's dedicated bandwidth. They are never over-subscribed.
I'm almost positive that this is not how they do things on the consumer side. Otherwise the Netflix debacle wouldn't be happening. But basically the consumer side is always over-subscribed and by a reasonably large factor. They're counting on consumer traffic to be bursty which it normally is. Netflix isn't bursty the way they're used to.
I sympathize with the idea of expecting one thing when building a model and seeing it play out differently. It sucks and it can cost a lot of money to make good on promises that you've made. But remember that Verizon is the one that made these promises. Their customers didn't put a gun to their head and say "promise me these crazy high download speeds or else!", Verizon did it willingly to try and steal customers from their competition.
Because they did it to themselves I have little sympathy.
See first page about CDN for a description of how they moved from CDNs paying for access to demanding ISPs installed thier CDN for free. http://arstechnica.com/information-technology/2014/07/how-co...
> But when we ask them if we too would qualify for no-fee interconnect if we changed our service to upload as much data as we download -- thus filling their upstream networks and nearly doubling our total traffic -- there is an uncomfortable silence. That's because the ISP argument isn't sensible. Big ISPs aren't paying money to services like online backup that generate more upstream than downstream traffic. Data direction, in other words, has nothing to do with costs.
> in other words, moving to peer-to-peer content delivery
Verizon is selling me a 50Mbps connection, but if I use it, for even a little, they get upset and say they are over capacity. This is the same company that over-subscribes their network on purpose, as most ISP's do. They will come into a neighborhood with a 1Gbps link, then sell 50Mbps to 100 homes, in hopes that not everyone uses the connection at the same time, nor for long duration.
The marketing papers over this in misleading ways, and that is bad, but if you wanted to pay for your own dedicated bundle of wires, like a businesses office building does, you could get it.
>The marketing papers over this in misleading ways, and that is bad
It's not a weak argument at all and the misleading marketing is exactly the reason it is not. In fact, Verizon has oversold bandwidth (or at least offered it for less than they are willing to accept).
But, what you seem to be effectively saying is that customers are at fault for buying an advertised product which touted the benefits they desired at a price that was acceptable.
If your last mile network was built to allow you to saturate your 50Mbps connection 24/7 your bill would be much larger than it is now. So much that you probably wouldn't pay for internet service, so oversubscribing is a totally normal and rational process when building out a network.
1. Kansas City, KS
2. Chattanooga, TN
3. Provo, UT
4. Austin, TX
These places all have fiber to the house, and supposedly gigabit speeds. Two of them were built out with zero help from Google, so Google can't be making huge losses on the internet to prop up other parts of their business.
> If everyone on those networks were saturating their connections 24/7, then yes, there would also be network performance issues.
Okay, first it was the last mile that was expensive. Now it's the back-haul that's expensive.
You do realize that you're contradicting YOURSELF here, right?
Your network no matter the bandwidth to your house is oversubscribed. If everyone was using their network connection at max capacity 24/7 there would be network congestion issues.
Thus the arguments that "I've been sold 50Mbps/down I should be able to use it at max capacity 24/7" without issue are incorrect.
No, not necessarily. That's how Verizon Residential chose to build out their network but that's not the only way to do so. The commercial side of Verizon manages to provide their customers with 100% of SLA throughput 24/7 no problems. Technically this is feasible.
I would agree that I might not be able to get 100% utilization at the price I am currently paying. That is a point I will concede. BUT, SOMEHOW Google Fiber in three cities and Chattanooga is also supplying gigabit fiber. And they're not charging $10k/mo per person to do so. Could everyone use the full gigabit connection 24/7? I seriously doubt it. But if they're oversubscribed by 10x on gigabit and other places are oversubscribed by 10x on 100 Mbps then they are successfully delivering 10x the internet for the same price, or perhaps 50%-100% more. It's not linear that's for sure.
For your argument to hold water if Google Fiber was 10x faster than Verizon, then Google Fiber would also have to be 10x as oversubscribed as Verizon. I very, very seriously doubt that this is the case.
> Thus the arguments that "I've been sold 50Mbps/down I should be able to use it at max capacity 24/7" without issue are incorrect.
This is a straw man argument. You are not engaging in an honest dialog. You seem to be purposefully obtuse. People watching Netflix in NO WAY constitutes 100%, 24/7 utilization!
How about "I've been sold 50 Mbps service and I want to use 3 Mbps of it for an average of 3 hours per day, which is a measly 6% of the advertised speed and is only 12.5% of the 24/7"
There are two ways to do the math here. One is based on peak utilization, where we assume that everyone watches Netflix at the same time, 100% identically, say from 6pm-9pm. In that case Verizon can be over-subscribed by a factor of roughly 16x (50Mbps/3Mbps) and still keep their customers happy.
The other is to assume that everyone watches Netflix perfectly randomly. In that case let's assume 3 hours per day. Here Verizon can be over-subscribed first by the 16x (since people don't use all 50 Mbps) and further by another 8x (since people are using the service for 3/24 hours per day). That gives you an over-subscribed factor of 128x.
The truth of course would lie somewhere in between 16x and 128x. But what we're seeing is that Verizon is over-subscribed by a factor of more than 16x otherwise people wouldn't have any problems watching Netflix. Put as percentages instead of ratios, customers are asking Verizon to provide between 0.75% and 6% of what they promised to deliver, and Verizon is refusing to do even that.
Further we have evidence via a blog post from Verizon that their entire network is NOT oversubscribed by a factor of 16x as it's only at 40-60% utilization. What is happening is that Verizon is throttling their users at the peering point. It's a great gig if you don't get caught. http://publicpolicy.verizon.com/blog/entry/why-is-netflix-bu...
If we assume that 100% of all traffic on Verizon's network is from Netflix that means Verizon could double the peering bandwidth to L3 before their network got congested. Is that a valid assumption? I don't know, but I sincerely doubt it's 100%. During peak hours it might make up 50%. In that case doubling the peering bandwidth would increase utilization from roughly 50% to roughly 75% which is perfectly acceptable.
It must take some pretty fantastic mental gymnastics to convince yourself that a provider who is not able to make good on 6% end-user utilization is totally in the right and their upset customers are irrational and wrong.
Both me and my brother get billed when we call each other on our cellular phones! Help me!
Also, your phone provider does get charged for each incoming call. They just eat/redistribute that cost, since for normal phone user it's likely to be less than a penny per month. But try running a conference service and you're definitely going to get billed for incoming calls (apart from some weird force-subsidised areas where you can get paid to receive calls)
From a Singaporean perspective, that's the most bizarre part of mobile service in the US. I need to get an unlimited texting plan so that I don't get billed for people sending me texts? Whoa!
I'm pretty sure you can still do this.
When I sign up with an internet provider, I am agreeing to pay a fee for open access to the internet at large. I'm not signing up to pay for each piece of content I access or each site I connect to.
I've never agreed, to the best of my knowledge to pay for access only to content served at a rate limited by the reverse utilization of the peering connection that content provider's content can be accessed across.
Are you saying that in the US when Ann calls Bob that both Ann and Bob will be charged?
If you are using a cell phone or a satellite phone, you pay for your connection to that network, whether or not you started the call.
You certainly don't make it so that grandma using her POTS has to pay for the guy with the more expensive connection. That is viewed as extremely unfair.
OTOH, using the cell network generally has free calling to anywhere in the lower 48. It's not the old physical wires that are the expensive part, it's maintaining and upgrading all the cell towers, and so people pay for that usage. Once you've paid to connect your call to the wired network, you can do whatever within very wide geographic borders.
http://devnull-as-a-service.com
Imbalance fixed, but I don't think that is what Verizon wants.
Of course, since the NSA would need to listen in on all that new random-bit traffic (since Netflix is obviously the new Skype in the terror playbook), taxes would need to rise significantly in order to pay for more NSA bulk-data-collection servers. But that's just the cost of living in America...
Seems to me it's a win-win for everyone, including poor border routers. Except Verizon then wouldn't make an extra penny from that efficiency, hmm?
Why should they have to do that? Level 3 has open ports now (so they claim). It doesn't matter which provider the traffic is coming from, either Verizon wants it or they don't (and apparently they don't, even though their core is no where close to saturated).
The balanced traffic requirement is bogus anyway. Its not as if Verizon offers their own video streaming service that I can subscribe to as a Comcast or AT&T customer. So they're never going to see balanced traffic in the age of video streaming.
- If you push traffic to my network it is because someone in my network requested it. - Incoming and outgoing traffic are technically no different. - It makes more sense to pay to receive traffic than get paid to receive traffic. That's actually what VZ customers pay for and that's why not many people complain about ADSL speeds.
If I were Level3 I would actively partner with VZ alternatives in the US, announce it to VZ customers so they can move away and then proceed to remove any peering with VZ. Enough bullshit already.
Instead of double billing (i.e. subscriber + content providers) couldn't Verizon just increase their subscriber rates to cover their perceived increased costs?
Could Netflix pay Verizon's ransom and then charge Verizon customers a "Verizon Tax" of sorts? Claiming that Verizon charges Netflix at extra e.g. $1/subscriber/month to peer?
Would it be legal for Verizon to literally just cut off Netflix? Just return an access denied to their subscribers.
For (2), Netflix could almost certainly do that, but I assume they'd prefer to not pass any cost onto consumers (because, really, they're making a stand against this business practice by Verizon being fair play at all). Among other reasons, if it's seen that Netflix is willing to pay a "ransom," other carriers may try to increase their contracted costs.
1. Verizon, as a monopolist, is already pricing its service at the level that extracts the most revenue while remaining within the envelope of whatever loose regulation they have.
2. Netflix, as a company in a competitive market, can't easily increase prices without losing customers. If Netflix charges Verizon customers $9 and Hulu is only $8, it makes Netflix look less attractive, and they'd lose customers.
Further Netflix isn't the only place to get movies. I can get movies from Amazon (streaming or DVD) Walmart, and other physical stores. I don't have that same kind of choice in ISPs though.
IPSs have more of a monopoly on their subscribers than video intermediaries do. The studios can release movies through a half-dozen channels. I would (figuratively) KILL for the chance to pick from a half-dozen ISPs that have offerings which are reasonable for the modern age.
We think of ISPs as selling us data, but the big ones have realized they can make more money as data market makers. This whole kerfuffle is just them trying to see if they can get away with it. Because they're effectively monopolies on the last mile, it's important that they fail here.
Verizon already increases prices x% per year, so maybe. But for every price increase there are some number of customers who will cancel or downgrade to a lower plan. (Alternately, Verizon could decrease their profits.)
Could Netflix pay Verizon's ransom and then charge Verizon customers a "Verizon Tax" of sorts?
Sure, but some percentage of Netflix customers would cancel.
Would it be legal for Verizon to literally just cut off Netflix?
Considering that the FCC is debating Net Neutrality right now, that would be a very bad idea.
There's more to "the Internet" than just "the WWW". While HTTP could be considered a "pull data" protocol, there's plenty of application protocols which involve "pushing" data over TCP/IP transport.
In other words Netflix doesn't start streaming data to me for the hell of it. Pandora doesn't stream songs to my computer unless I hit play. Songs don't get downloaded from iTunes unless I request them and pay for them.
This is in contrast to the mail that the USPS delivers to my door. I don't ask for most of it and I have to sort through it and throw all the bullshit away.
I guess what I mean is that most internet traffic is based on consent of some kind. While paper mail is not.
I might not specifically request any particular bits but the majority don't show up unannounced and unwanted.
> there's plenty of application protocols which involve "pushing" data over TCP/IP transpor
Right but by the time a TCP link is established you've already setup a session which can't be done unilaterally. That is akin to consent and quite possibly a request. You can't be "data-raped" because if you decline the TCP session that's the end of it. Someone can send a billion request to initiate TCP sessions but they can't initiate without your (or your computer's) consent.
Given the router/firewalls that basically everyone has between their modems and their computers it's getting very, very difficult to send data to residential ISP customers without their express or implied consent.
But under NO circumstances could you claim to do UDP over TCP/IP.
Were Netflix performing this "data rape" of Verizon's network it would be huge news as a DoS/DDoS attack. But that's not what we're hearing.
CAN people perform attacks? Yes absolutely and it requires no consent. But without some kind of consent it's regarded as an attack rather than as legitimate network traffic.
Yours is probably the clearest explanation I've seen of why this all feels so icky.
As an example I use a website (like Facebook or something) which does live auto-updating. Under the hood it's something like WebSockets or SocketIO and the server does send datagrams to me without me specifically requesting those individual data packets. I couldn't request them, I didn't even know about them until they arrived!
But while that technically is "push" from a high level it's still "pull" in that Facebook (or whoever) doesn't just start sending them to me for giggles. It doesn't start happening until I visit their site, and it stops happening fairly shortly after I navigate away from it. Their computers don't unilaterally send packets to me. I request them by continuing to execute the javascript that their servers sent to me when I loaded the page. Once I stop executing it, the connection is closed and the messages stop.
Uh, I don't think he meant it the way you took it. I think he knows how network traffic works.
To be pedantic curl -d test example.com would push data across the internet.
If it doesn't accept the connection your push goes nowhere. Imagine the webserver is down or the port is blocked. In that case the TCP session doesn't get initiated and you're unable to send the data.
http://en.wikipedia.org/wiki/Transmission_Control_Protocol#C...
EDIT: Considering that most consumer ISPs don't allow their customers to run servers (Google Fiber is an exception) it would be very very difficult for someone to successfully "push" data to consumers. And that's only if I'm willing to accept your definition of push, which I'm not.
Also, that has nothing to do with the point that "pushing isn't a thing on the internet" is a poor way of describing the problem being described, especially since the poster said the internet and not consumer's using their ISP (which still isn't strictly true, but is at least closer to the idea of the asymmetry in most consumer and business interent connections).
That doesn't negate the point that consumers download much more data than they upload.
When I say something incorrect, I expect to be corrected. I attempted to correct something that was factually wrong and don't believe I was mean while doing so.
I don't have any hard numbers but I would guess that at least 90% and very likely 95% or 99% of all internet traffic is pull-style.
Transferring data between two parties on the internet for any reason usually goes hand in hand with one person wanting to receive some data and another wanting to send it. If the sender doesn't want to send, no transfer takes place. If the receiver doesn't want the data, it's regarded as an attack.
I am having a hard time coming up with legitimate situations where someone sends data to my computer without my permission (express or implied) and that doesn't seem like some kind of attack.
But I'm also open (at least I'd like to believe I am) to someone cooking up a situation that proves me wrong.
Also, nothing I said despite it being asymmetric, just that his point about pushing and pulling isn't the best way of describing the issue.
His point about pushing vs pulling is a perfectly fine way of describing the issue. It's limited, but all explanations are.
Not always. Backup is popular today - as I look at my simple home utilization I push about 30% of the ingress up to "somewhere".
> Double billing for a single service is a neat trick if you can pull it off.
This is how every communications network has been ideally built from the beginning and this "trick" has been played out time and time again starting starting with the PSTN. It's not uncommon, but is also isn't ideal - unless you are the carrier.
You're equivocating on 'push' and 'pull'. Under OPs definition, a 'pull' is a customer's valid usage of the service, while a 'push' is something like unwanted or unrequested data transfer from a 3rd party. This definition is a little muddled when you talk about uploads but it should be easily understood. You uploading your backup data is you merely using the service you paid for, even though you're 'pushing' data somewhere.
No. That's not how the internet works. Nobody PUSHES data. People PULL data.
Verizon has customers who pay for internet access. Those customers make REQUESTS to Netflix for data and Netflix RESPONDS with data.
They happen to be streaming movies which use a lot of bandwidth, but on average it's about 3Mbps per concurrent stream. That is WELL below the 25/3 or 50/5 or 100/10 that Verizon advertises for purchase.
If someone were pushing data it would be called a DoS or DDoS. An attack is when someone sends unrequested data to try and break your network.
But this data isn't unrequested. Verizon's customers have requested it from Netflix as they are within their rights to do since they have literally paid for it.
This is Verizon wanting to bill their customers once and then their customers' vendors as well. Double billing for a single service is a neat trick if you can pull it off. But it tends not to engender goodwill.