> This is what happens when you treat IT like a cost center and don't provide the necessary funds to tackle technical debt and keep your services up and running:

Especially now with cordcutting becoming the norm and traditional cable subscription number dropping, large providers are scrambling to find ways to extract profit.

With the rise of streaming, peering arrangements are more lopsided than ever and ISPs are actually having to pay for their own internet access.

Because of this we're going to start seeing a lot more data caps, neglecting basic network infrastructure, and other tactics to extract as much profit as possible from customers.

> With the rise of streaming, peering arrangements are more lopsided than ever and ISPs are actually having to pay for their own internet access.

Don’t most ISPs cache this sort of content closer to their customers? Netflix at least offers to install hardware that does this.

YouTube and Netflix cache boxes are important for smaller ISP's. Their cache performance can literally make or break the monthly P&L. That said, I haven't heard about other OTT services offering a caching appliance. And there are more and more OTT streaming services coming. It's great from a consumer perspective, but it's going to be challenging for smaller players.

Moving towards decentralised internet infrastructure such as https://ipfs.io could help with this. ISPs setting up IPFS caches for their customers could save a LOT of peering bandwidth.

Solution is to get some datacenters on your network as well. I'm not sure why that didn't happen before.

How would a rural WISP "get some datacenters" on its network? Sorry, I'm not grasping your previous comment.

If you have the problem that peering is getting very one-sided, the solution is to get the other side as well. So a rural ISP should try to expand into an area with cheap power and offer low-cost hosting and bandwidth to attract servers.

This is downright silly. The bulk of bandwidth is associated with a select few destinations that aren't going to put their datacenters behind a consumer isp under any circumstances. But many do offer caching options that allow content to be sourced inside the ISP.

Hard to do that when everyone runs everything in the cloud. That's a very tough market to crack into.

I'm not sure... hell WB will have 3 services... WB/CBS/DC-Universe etc... Netflix, Hulu, some TV alternative (Hulu Live, Sling, etc). It all adds up, and may not be a value. I swear, if the GF wasn't hooked on Bravo, I'd give it all up and just pirate everything again. I honestly do some shows just because it's easier to use my seedbox and grab everything new that I watch once a week or so, and work through the new episodes.

> This is what happens when you treat IT like a cost center and don't provide the necessary funds to tackle technical debt<

Cost-cutting didn't seem to have played a role in this outage. They had a bad linecard in their out of band network that was somehow impacting the control plane of their devices. You can talk the pros/cons of how they built out their national Out-Of-Band (OOB) network, but you can't blame this outage on cost cutting.

You're simply bashing a large company that had a bad day. They might deserve your vitriol for plenty of other reasons in other cases, but you can't blame this outage on cost cutting.

It seems to me that they clearly had some serious problems with their overall network architecture if a single bad card could bring everything down. If they'd invested more money into IT over the past years then they could've taken care of a lot of their technical debt proactively rather than reactively. Odds are decent they would've implemented bad packet filtering before the major outage instead of afterwards, as that seems like a plausible thing an IT/networking department would've gotten around to with more funding, and more and higher-skilled, higher-paid veteran staff who could pre-emptively identify and fix this issue through experience and testing.

Any time something really bad happens that could have been prevented, it's reasonable to ask why it wasn't prevented. This wasn't some black swan event, just a routine hardware failure that they didn't have defenses against.

Who's to say that it's technical debt? Maybe they liked the way that their management network is designed? It's only debt if you realize it's bad and ignore it.

To be perfectly honest, I don't know what's the best choice for running a management out-of-band network for a network provider of this size. Do you?

We don't, and apparently they didn't either. But it was their job to do so.

I think it's very reasonable to suspect underinvestment and technical debt here. Not only did they have a problem, they had a very hard time a) finding the cause of the problem, and b) mitigating the problem while they were looking for the cause.

We can't know, of course. But I think it would be hard to argue that this is optimal, and that nobody at CenturyLink ever suggested it could be better if they spent a bit more money.

It's hard to work on network problems when the management network is also having problems. That requires dispatching people locally to sites to see what's going on. You're blind in both eyes, not just one.

Look, I'm not defending them or their choices. I'm just saying that this more more nuanced than "They suck they should have known better or spent more money." When you run a network as large as this and with as many "legacy" technologies as they do, things aren't very cut and dry.

NTT probably has the best managed global network with their extensive use of SDN. They run a tight ship and everything goes through their automation frameworks. This requires a huge investment in R&D. Even then, it doesn't cover 100% of their network because "there's that T320 for that customer in Chicago, and the stuff up in Michigan for Dorian's T1." Multiply that times 30 or so to consider size differences between NTT and CenturyLink....and yeah, stuff happens.

I agree it's hard. Anything interesting is.

But your last paragraph specifically describes technical debt as part of the problem. If even the best org (NTT) has technical debt and CenturyLink isn't the best org, then I think it's safe to suspect that a) CenturyLink has significant technical debt, and b) they have underinvested in that technical debt compared with industry best practices.

This issue is almost the same issue I had when I had an outage that took down the internet connection for a local govt I worked at.

The issue was a single bad port on a switch about 3 hours away. It took two days of downtime for them to figure it out.

Now the place I work at just deals with having fibre cut 2-3 hours away and losing a days worth of work for it. At least now I don't have to rely on insiders to tell me what actually is going on. But Centurylink does offer a 4g service for backup connections....

I'm not surprised at all with this. Business as usual for Centurylink.

Honestly, if you had FiOS as an option why would you stay with CenturyLink in the first place? Verizon is just as assholish, but at least they can deliver.

At least in my area Century Link business is the only isp I can get at my house that doesnt come with a data cap. Both CenturyLink home and our cable provider have data caps that are enforced. And working from home along with Netflix and YouTube blow me through the 80gb limit every month. Causing me to have to raise my plan. And I’m not paying $200 a month for unlimited bandwidth.

Your cable options have an 80gb limit?! WTF!! Comcast has data caps (which is stupid) but it's 1TB and I think my 7 person household has only passed the halfway mark once, so it's meh. But 80GB?! That's insane! Destiny 2 is an 80GB download itself.

I live alone, and generally I do about ~400GB of data transfer on Comcast a month. I work from home, and spend a lot of time downloading Docker containers and other such things to do my work.

Last month apparently I hit a new high for me: 890GB. Lots of devices needing updates/games downloaded...

I am really surprised that with a 7 person household you don't go through your data even faster.

I even run a local caching server for all my Apple devices, which over the last 30 days has saved me about ~50GB from having to be served from origin, with about ~100GB being served to clients.

Although I tend to also be a heavy user of Apple's iTunes and particularly their movies service, so I have a feeling a lot of that cached data is me putting movies on in the background and them being streamed from the cache rather than from Apple directly.

Any option to host your docker stuff on a VPS or other system that isn't using your home internet connection?

I think he/she uses Docker as a development tool on his machine, with volumes and everything.

Yep, this isn't about hosting docker containers or the end result, but doing development work/tearing down/setting up test environments and things of that nature.

It is surprising how big docker containers get after a while.

Seems strange how bad ISP's are in the US compared to the UK.

I can get 400/35 Fiber to the Home with no datacap (no throttling either) for £54 a month (~$70).

> Seems strange how bad ISP's are in the US compared to the UK

UK average fixed-line download speeds in Q4 2017 were about 26 Mbps [1]. The same Q2/Q3 2018 statistic for the United States was over 96 [2].

American broadband is crappy. But in mean technological leadership, it’s ahead of the UK. (At the leading edge, I get 400/35 for $80 in Manhattan.)

[1] http://www.speedtest.net/reports/united-kingdom/#fixed

[2] http://www.speedtest.net/reports/united-states/

I wish they'd do the median rather than the mean. Having a small percentage of people with super high speeds will drag the mean up, but it doesn't change the experience for the typical consumer.

Bill Gates walks into a bar. The mean net worth of bar patrons is > $1,000,000,000.

I'm in Manhattan too, and with FIOS my options are 100/100 for $40, 300/300 for $60, or 940/880 for $80. Admittedly those prices might be first year discounted prices.

What's really strange is how the crappy broadband is continuously justified in the US.

Replace “broadband” with everything from education and healthcare, through infrastructure and a dozen other words, and you’re still right. I don’t understand it either.

Education and healthcare are of high quality in the US, generally speaking. They are just super expensive.

The Radio Eriwan joke writes itself.

Oh man, those still hold up too. “In principle yes, but unfortunately nobody here has the education to be sure.”

The belief in American exceptionalism isn't limited to just positives.

Broadband in the US varies wildly - I'm paying $85 (~ £67) for an 800/800 fiber connection, but I am in a market where Comcast and Centurylink are slugging it out, so we have much better offerings than many parts of the country.

Depends on where you are. I’m paying $80/month for 940/940 fiber with no cap or throttling. A few miles down the road the only option is slow and unreliable cable through Comcast.

I don't know if it's widespread but at least here in WA you can pay Comcast an extra $50/month and they'll remove the data cap.

Comcast Business class accounts don't have the cap as well. It's slightly more expensive and comes with slower speed tiers but the support has been better. It's not required to be a business to go through that channel.

Comcast business doesn't have caps, I pay about $90 a month.

This is why amazes me. At my home we blow through on average 25 GB a day in gaming, streaming video, other downloads etc. limits like these are terrible. 80 a month? Wtf?!

25 GB a day? Are you streaming 4K?

I live alone and use 50-100 gb/day on days I’m home all day, so 25gb/day isn’t that much.

I cut cable TV years ago and exclusively stream. I like having the TV on for background noise when home, even though I’m usually not actively watching it the whole time, and you’d be surprised how fast one can chew through data usage. I usually use around 1.5TB a month when you factor in offsite backups.

30gb to 1.5tb. And the last one I had it I was paying over $100 a month for 150gb.

https://support.cableone.net/hc/en-us/articles/115009159707-...

I'd say the majority of consumers stick with their existing suppliers out of sheer inertia. It takes time, effort, and hassle to switch, and absent any meaningful change in status quo (like a prolonged outage or sharp price increase), most people will coast. Especially when they're not technically inclined, as the difference in service will be less meaningful or apparent to them.

FiOS in some parts of the country (including, I think, Oregon) is now run by Frontier, who are nowhere near Verizon's competence.

centurylink fiber is pretty great as well

Yeah, my experience with Clink's fiber offerings has been a good one. It's definitely "Gigabit" — there are many fiber splitters in play, which means (a) your downstream traffic is broadcast to everyone on the same splitter network in plaintext and filtered at individual ISP-owned termination devices, much like cable networks, and (b) you definitely never reach 1000 Mbps down, but often can reach 900 Mbps up. Typical is 500-600 Down, 900 Up; the highest I've seen down on e.g. fast.com is 900 Mbps. And the price is extremely competitive with Comcast ($80/mo for "gigabit"), unlike their DSL offerings, which are complete crap.

The major downside is that their IPv6 support is nonexistent and they have no plans for rolling it out.

I've been getting perfectly adequate IPv6 via their 6rd gateway (on fiber, in Seattle). I wrote some nerd notes on using one's own wifi router and setting up IPv6: http://b.tra.in/2015/07/notes-on-centurylink-fiber.html

6rd is pretty awful, generally, and I couldn't get Centurylink's to work last time I tried. I appreciate the notes and will take a look, thank you.

(One big problem I've experienced in the past is that any time you configure both IPv6 and IPv4, all programs prefer IPv6. But tunneled IPv6 has much worse performance than the native IPv4, so you really want applications to prefer IPv4 in a tunneled v6 environment.)

I still use Hurricane Electric tunnels fir my v6 and ping times/throughout are so close as to be in the noise for most everything. It probably helps that I'm using a tunnel endpoint literally 10 miles from my house however.

My IPv4 ping times to Google and Facebook are 2.6 milliseconds. I suspect any HE tunnel is going to be worse than that (in a statistically significant sense). I don't know that latency would necessarily be perceptible the majority of the time, but I strongly suspect outliers would be worse and more frequent.

Also, I doubt HE's ipv6 tunnel is going to carry anything close to 1Gbps for me for free ;-).

You sure it’s plaintext? GPON networks should be using AES128 to each ONT.

No, not sure. I wasn't aware of that, thanks. Do you know more about the key negotiation and encryption protocol? Wikipedia doesn't have even mention AES. Thanks.

You do know that perfection with TCP over Ethernet (no 802.1q rages, no retransmits) at 1gbps is 941Mbps, right?

What is your point? All of the observed numbers I've provided are sufficiently well below that as to make the distinction between 941 and 1000 meaningless.

Come on, folks.

> Typical is 500-600 Down

500/941 vs 500/1000 is 50% vs 53% of advertised. The 3% isn't really significant. I don't think it's bad service for the price, but it certainly isn't true 1gigE.

Here’s what you wrote

> but often can reach 900 Mbps up. Typical is 500-600 Down, 900 Up; the highest I've seen down on e.g. fast.com is 900 Mbps

Yes, I wrote that as well. What comment are you making?

Typical observed speeds down, which is what everyone cares about for residentical internet, are 500-600 Mbps — far below 1Gbps. I mentioned the outliers for color but they're either not useful to me (I don't need 900 Mbps upload) or not representative of real service delivered (I've only observed 900 Mbps once; as I said, 500-600 is much more typical). Off-peak is typically around 700 Mbps down.

So I still don't know what comment or point you were trying to make by claiming GigE TCP can only reach 940 Mbps. It clearly isn't a counterargument to my claim that Centurylink's $80/mo, residential "gigabit" fiber offering does not deliver 1Gbps internet service (nor would I expect it to).

Hi, I'm on centurylink's fiber offering and have the same experience as you getting much faster upload than download (though download speed increases quite a bit during sleeping hours). Are you saying this is because the DL is congested during the day hours? Or is there some other technological block in play?

> Are you saying this is because the DL is congested during the day hours?

Yeah, it's congestion on the transmitter (upstream) end.

IMO the plaintext part isn't a real issue. With the recent dramatic increase in https usage it won't have much impact.

I'd also consider that segment "comprimised" anyway, from a security perspective anything past my modem could be MITM'ing my connecting.

Yeah, I totally agree; at the end of the day end-to-end authenticated encryption is the way to go. But not everything is end-to-end encrypted, so it's worth mentioning as a caveat. It sounds like I may be mistaken anyways and some GPON networks negotiate an AES key for downstream traffic, but I have no way to confirm that this design or implementation is secure, or that in practice CenturyLink even uses it on my local fiber.

Doesn't exist across 99% of their last mile footprint. Just one or two cities.

I would guess the vast majority of their revenue comes from extorting old people and farmers with shitty overpriced ADSL that squirrels eat into and take down annually.

Rural folk here. Biannually.

Here in Denver we get full 1Gbps for $75/mo, every provider has outages and this was the first partial outage we experienced with the service (pages loaded just slowly most of the day)

If only it was available in Golden... I could hit the CL POP with a baseball (less than a block away).

To be fair, this does seem like an extremely rare root cause, but yes the impact was awful, so it'll be worth noting how CenturyLink responds and what they'll do to try to prevent similar issues in future.

At scale, rare errors will occur on a semi-regular basis.

Multi-day outages tend to be indicative of a dysfunctional internal organization.

Totally fair - I don't mean to excuse the results. That's an interesting insight that perhaps it's possible to identify organizations who have scaled (or whose products have scaled) too quickly based on whether they can cope with outages.

Just deciding they want to put it up 50 dollars a month is insane!