This is one of the underreported issues in the whole net neutrality debate. In the late '90s I worked in the WAN engineering group doing capacity planning at a big regional CLEC, which actually was a Tier 1 backbone provider. It was widely assumed in the industry that the flat rate pricing model -- pay $X for Y "guaranteed" bandwidth -- was living on borrowed time and would be replaced by actual usage-based pricing, paying for packets like you pay for gallons of water. The flat rate model only worked because we generally could keep available capacity well ahead of demand. We could do that because we were adding customers like crazy, and customers sucking down high bandwidth were outliers. If those prerequisites ever changed we'd be screwed.

Well, here we are in 2014 and ISPs like Comcast are not adding customers like crazy anymore, because most of the people who want high-speed data from them already have it. But the customers they do have are, on a per capita basis, using much more bandwidth now than they were five years ago. And this is only going to get worse. The nightmare scenario is basically upon us: capacity requirements are climbing faster than ever, but those requirements are no longer being driven by customer acquisition. Which means either the ISPs eat the cost of upgrading their infrastructure (ha!) -- or they raise rates.

But wait, doesn't Netflix pay for all their bandwidth already? Well, probably not: unless industry practices have changed drastically, even those backbone trunks are actually oversubscribed. If you pay for (say) a DS3 line, there's an implicit assumption that you are not going to be pumping 44.736 Mbit/s through it 24/7. Netflix breaks that assumption.

I don't know quite what the solution here is, and I certainly don't want the kind of dystopian "the ISP only gives you access to web sites that you pay for" future the sharpest critics paint. But I don't see this as a step toward that dystopian future. What I see it as is the ISPs trying to figure out how to work usage into pricing models without saying "screw it" and actually adopting a usage-based pricing model.

If it is really expensive to accept Netflix's offer of free hardware and free bandwidth then the additional cost should be passed to the customer. The problem is that it's not actually more expensive and Comcast knows their customers can't choose someone else so it's a much better strategy to squeeze the content providers.

Let's do some simple math to prove your point.

Assume Comcast has about 10 Tbps of traffic at peak, single-counting all traffic that transits AS 7922 (Comcast's national backbone network). [1] Now assume that a full 50% of that traffic is originated by Netflix, and the market price for transit is $2/mbps. The percentage is likely quite lower than that, and given 5 Tbps of traffic volume and coordination between the two networks to reduce the number of miles the bits need to be hauled by both networks, the per-megabit price is almost certainly quite a bit less.

But even with these obviously flawed guesses, we're only talking about $120mm/year in costs. Netflix's worldwide revenue in 2013 was $4.3b [2]; Comcast's 2013 revenue from their US HSI business alone was $10.3b. [3] If 50% of Comcast's $10.3b business was in jeopardy, don't you think they'd find a way to absorb that $120mm?

[1] http://as7922.peeringdb.com/

[2] Netflix Q4 2013 earnings release, http://ir.netflix.com/results.cfm

[3] Comcast 10-K filed 2/12/2014

(edited: formatting)

Netflix could have held out, and made Comcast out to be the bad guy, or could have leaned on Cogent to get the circuits installed - but, in this case, they folded and just decided to pay Comcast. This will probably set a very bad precedent for them when it comes to renew those agreements in 5-10 years, or however long they will be. Maybe they are hoping for some competition to enter the US environment by then.

How would that work? Comcast customers by and large can't switch to another broadband provider and Netflix customers can certainly switch to another video provider (Comcast!). It's not like this is a brand new issue, Netflix has held out for a long time. The Comcast / TWC merger is what finally did it, 30M combined customers is too big to risk on a public game of net neutrality chicken.

Surely they're still working on lobbying efforts to make it illegal for a company like Comcast to simultaneously have a monopoly on selling broadband and compete with services using said broadband. It's pretty obvious at this point that internet access is a utility and it should be treated as such.

The Hacker News readership includes many people capable of starting ISPs. Comcast's large profits, which have been used to fund a string of high cost acquisitions, are a juicy target for potential competitors. Yet you never see discussion of ISP startups on Hacker News because the barriers to entry protect the current carriers. This wasn't the case in the 90s, and had Hacker News existed back then ISP startups would have appeared frequently.

The issue of telecoms not having competition has been the case for years. Back in the BBS days Jack Rickard used to harp on this in Boardwatch [1] magazine. I recall him saying something like, "Two guys with a screwdriver should be able to start a phone company."

[1] http://en.wikipedia.org/wiki/Boardwatch

Indeed. The late 90s / early 2000s were the golden age of the CLEC. There were certainly pain points (like getting copper pairs provisioned), but I remember not caring that my building didn't support Comcast Internet because I had an SDSL connection from Telocity and an ADSL connection from Speakeasy.

That said, most were either bought out, or went under within a couple years. It was a rough time for commercial SysOps in the mid-90's... not that I really feel bad for them. Times and environments change... The BBS and art scenes are still around today, but not nearly what they were in the early-mid 90's...

I miss it a lot... since most BBSes were based in a local area, there were a lot of get togethers, and you'd meet and talk to people you wouldn't necessarily do in today's internet, which is based in interest groups.

The cable industry has evolved into a bunch of regional monopolies that do not compete with one another.

The effect of breaking Comcast up AT&T-style would be... a bunch of regional monopolies that do not compete with one another. And then, as with AT&T, proceed to consolidate via M&A into larger regional monopolies... that do not compete with one another. Same as now.

Welcome to the American telecommunications industry.

""" But there's a silver lining for people who hate this deal: By the government's own standards, the Comcast—Time Warner deal should be blocked. To understand why, you have to know that there's a thing in antitrust law called the Herfindahl-Hirschman Index, or HHI. In very basic terms, the HHI is a way to measure the concentration of a given market. And the HHI is the test the government's lawyers apply if they want to see whether a proposed merger would create monopoly-like conditions. """

That's totally backwards. Comcast's customers are paying for access to Cogent's customers, including Netflix.

It's not Comcasts responsibility if Cogent did not have peering arrangement or had paid for sufficient capacity to handle their end of the connection, that their customer was paying them to provide.

If it was, I'd love to provide Netflix connection much cheaper than Cogent. I'll provide it over my 10Mbps uplink, and blame consumers ISPs when everything falls over.

Since you've paid Comcast, they're now required to go to my brother's ISP and arrange more peering, right? Also, since you're paying, I shouldn't have to pay my brother, and he shouldn't have to pay anyone else, right?

Come to think of it, why isn't all hosting bandwidth free? Why do Azure and AWS charge for outbound bandwidth, but not inbound?

Netflix pays level3 for transit service (you pay your brother). Comcast pays level3 (your brother) for transit service. Notice how level3 is neither the source nor the destination of the traffic.

Netflix's traffic increases from, for example, 500Gbps to 1Tbps. Comcast's level3 ports fill up because netflix increased in popularity.

In general networks that pass a lot of traffic like this end up peering directly and avoiding transit costs.

Comcast has two choices. They can either pay level3 for another 500Gbps or they can transition the full 1Tbps to free peering directly with netflix. And yes, I consider it Comcast's responsibility to do one of those two things with their subscriber revenue, which, incidentally, most other ISPs in the US are doing.

Instead they are choosing option 3 which is, leverage their local access monopoly to force netflix to pay them, and protect their cable and xfinity services.

This may be the cases for ISP-ISP connections, but every transit circuit I've ever purchased as an end-user has two elements, a Non Reccurring Charge (NRC) to set up the connection, and then a monthly recurring charge (MRC) consisting of, a port speed, and a 95th percentile metered usage charge.

e.g. I might get a gigabit port set up for an up front $5000 NRC, and then a 24 month contract for $2000/month port speed, and then pay $5/megabit @95th percentile of 5 minute averages sampled over a month.

The ISP samples the usage of my port every 5 minutes, (8640 samples in a 30 day month), drops the top 5% (432 samples), and then charges me the cost of the next sample (sample #8208) - If the sampled speed at #8208 was 1 Gigabit, and sample #8207 was 0 kilobits, then I am still charged $2000/month + 1000 * $5 = $7000/MRC for that month.

Cogent has always had a reputation in the industry for being the cut-rate player (to the point of other ISPs just dropping them as peering connections, resulting in companies literally not being able to communicate with their customers) - Netflix was trying to rely on Cogent to send data to their customers, presumably because they were charging significantly less than "Tier-1" transit-providers, and finally came to the realization that yes, you do get what you pay for.

None of this surprises me, and none of it appears relevant to the conversation regarding Net-Neutrality - this is a peering discussion, pure and simple.

Each individual Comcast customer has in theory paid for a DS3 worth of bandwidth (modulo the "cap"), which should be a reality-check that some of your assumptions from that era need updating.

The defense (and it was always a thin one) of ratio clauses in peering agreements was hot potato routing. The ingress traffic is what you actually have to haul a long distance, so the agreement says "you must meet us everywhere we peer, and you must maintain ratio." Together, those rules mean each individual path will be asymmetric but the overall load on the WAN will be shared between the peers. It was always slightly silly, though.

Hot potato is now over. CDN's are desperate to do all the work and bring traffic as close to eyeballs as the monopolists will allow because the cost of actually building a network is irrelevant compared to the rent the monopolists are extracting. The monopolists are selling so-called "transit" to haul packets 1 mile. They're really charging for access to their captive eyeballs, not for moving bits. If you want to send a few megabits to Asia, fine, whatever. You can do it for the same price. They're not even mad, "glad we could make a deal," etc. The situation is nakedly broken and warped, and everybody actually in the business knows it.

Peering has always been a mix of market power, shaming, and political excuses ("ratio" was one). What we're now seeing is the shaming and excuses fall away thanks to apologist bloggers and weak politicians, and market power emerge as the only thing that matters. Compared to this power, traditional outdated notions of neutrality are irrelevant: there is no need to give traffic less "preferential" treatment when you can simply depeer them and then try to sell them so-called "transit" to go 1 mile. "Paid peering" == "well, how much money you got? Let's make a special deal just for you".

Without meaning to express either approval or cynicism, market power has always been the only thing that matters in this business. But CDNs have paid big ISPs special rates for years, because they really do put special strains on infrastructure. And my impression remains that that's what this particular battle is ultimately about.

I personally think the resistance to usage-based pricing in the tech community is really weird, although I mostly see it on the consumer end. Hard caps are bad, and caps that only apply to some services are really bad, but charging more if you use more is completely reasonable.

(Not to mention that there are no inherent variable costs to transporting bits. Water needs to be purified, electricity is obviously usage priced - but you can be pushing bits forever once you have the hardware setup.)

With usage-based billing, the amount will float up and down and it's hard to understand how the amount is calculated. Usage-based telephone bills are broken down by calls, which you can probably remember.("What's this 46 minute call to 555-1234? Oh yeah, I called Mom last week.")

With IP traffic, how do you itemize the charges? Temporally? I used to get cell phone bills that included pages and pages of stuff like "Jan 27, 4pm, 239789987345 bytes". It was completely useless.

How about reverse DNS? That has other problems. • You use BitTorrent? Here's 247 hosts with incomprehensible names on your bill. • You use a VPN? A whole bunch of traffic is lumped together. • "Honey, what's all this traffic to nastyporn.com?" • It puts pressure on web sites to optimize their deployment strategy for billing clarity

The "correct" way is probably to log everything, then provide analytics software. But I can't imagine my Mom generating reports just to understand her internet bill.

Economically, sure, usage-based billing is a great idea. Alignment of incentives and all that. But practically, it's a pain for everybody involved.

My electricity bill shows only X kWh per month. There is no "itemization" of usage. However I can purchase watt meters that can monitor power anywhere (including the main grid connection) if I decide the bill is too high.

I can definitely imagine a similar service sitting on the router or provided by the provider that does bandwidth accounting to deliver a picture of what is going on. I'm sure there would be many third party device manufacturers willing to do this traffic accounting as well.

Whoa, whoa, whoa. This is exactly the assumption commercial customers paying for unmetered links are making, otherwise they wouldn't be paying for unmetered links. Not being oversubscribed is supposed to the entire point. If ISPs' business models depend on them being able to break their promises to customers paying for continuously saturated links that's their problem, but they're still on the hook for what they're being paid to provide.

I dunno.. being profitable enough to buy time-warner cable for billions doesn't sound like a nightmare scenario from their POV. It sounds like they'd rather spend the money becoming a monopoly than improve their infrastructure.

Netflix does not send any data. Comcast customers request lots of data from Netflix. Significant difference.

Another example (just in the other direction): A backup provider needs to pay up big-cable because their customers are uploading a lot of data?

I'm willing to wager that backup providers haven't run into any upload issues with Comcast for this reason.

Generally speaking settlement free peering requires that the arrangement be mutually beneficial. If the traffic exchange becomes inequitable then the side that disproportionately benefits would need to pay to cover it.

In this case Cogent was sending vastly more traffic towards Comcast than they were receiving (because of Netflix) so they were the misbehaving party.

I'll admit my knowledge of peering is minimal, but consumer ISPs have always been something of an anomaly if memory serves. The traffic flow is highly one sided, but peering with consumer ISPs is obviously necessary if you want to actually reach consumers.

For example, they are the 30th largest provider globally going by the CADIA rankings (http://as-rank.caida.org/?mode0=as-info&mode1=as-table&as=79...) and have large traffic sources as customers. From the CADIA list this includes Peer1, Dreamhost, SoftLayer, Hivelocity, and Liquid Web.

As a eyeball ISP I occasionally get offered cheap transit from hosters looking to balance their traffic ratios. From the CADIA list it seems like Comcast may be doing the opposite.

When we do business with cloud providers or other third parties, we usually start with internet based access. When the relationship gets bigger or needs to serve a large portion of our base, we typically either peer with that provider (in the case of big cloud providers like Amazon, Microsoft, etc) or establish dedicated connectivity between our networks.

I think Comcast, Verizon, TWC, etc has a point here -- as a relationship grows with another party, you should have a more robust connection. Netflix hasn't wanted to do that -- it prefers to use a shitty ISP (Cogent) for cheap, and augment it by co-locating content on end-user ISP networks for free. Services like Netflix and Youtube stretch the net-neutrality argument, because they aren't good citizens.

That said, the end-user ISPs cannot be allowed to discriminate, which their monopoly power will almost certainly enable them to do.

Assuming it wasn't harmed is also speculation and we should definitely be concerned about the deal and what it means for net neutrality.