News for hackers. Stuff that matters.

Internet Issues in the US - Possible Backbone Issue

There seems to be an issue with Level 3 at the moment which is causing all sorts of sites to be unreachable for some people but not others. Rackspace (mail, hosting, cloud, etc), Facebook, Hotmail, Github, Digital Ocean, etc.

For those experiencing connectivity issues, try a different ISP/network. For example, I can't connect to any of the above networks via my regular Time Warner cable connection or via my T-Mobile hotspot. But I can connect via my Verizon hotspot.

( 46 comments | 88 points | on HN )
Re:Internet Issues in the US - Possible Backbone Issue by raiyu on Saturday October 19, 2013 @01:19PM (#6577053)
We can confirm that there are large issues on Level3's network we've had to pull them from the routing mix until they get everything resolved and stabilized.

Thanks, Moisey cofounder DigitalOcean

Re:Internet Issues in the US - Possible Backbone Issue by apaprocki on Saturday October 19, 2013 @11:32AM (#6576721)
Too funny -- I diagnosed this to L3 on my home net this morning and I found myself saying "I wonder if anyone else has this issue?". When it comes to the backbone providers there is no Twitter account or status.level3.net or status.theinternet. Hard to do when everything is decentralized. I usually go hit NANOG when stuff like this happens but you're not guaranteed to find something real time there.
Re:Internet Issues in the US - Possible Backbone Issue by busterarm on Saturday October 19, 2013 @12:26PM (#6576897)
Re:Internet Issues in the US - Possible Backbone Issue by pearkes on Saturday October 19, 2013 @11:36AM (#6576732)
Yea, same here. I took this traceroute (mtr): https://dl.dropboxusercontent.com/s/v5k29jnfkhz0ggu/2013-10-...

Not sure what was going on there, I wish I did, but seems to be related?

Edit: that was going to an nyc digital ocean VPS

Re:Internet Issues in the US - Possible Backbone Issue by malandrew on Saturday October 19, 2013 @04:40PM (#6577719)
Out of curiosity, what do you do to diagnose something like this? What tools and commands?
Re:Internet Issues in the US - Possible Backbone Issue by bcl on Saturday October 19, 2013 @05:49PM (#6578025)
http://www.internetpulse.net/ and http://traceroute.org/ are two sites I used in addition to the usual unix tools like traceroute.
Re:Internet Issues in the US - Possible Backbone Issue by throwaway2048 on Saturday October 19, 2013 @07:49PM (#6578426)
many providers also host "Looking glasess". Its a server hanging off their core network that allows you to run pings and traceroutes

http://lookingglass.level3.net/ (seems to be down currently)

a nice big list of looking glasses

http://www.lookinglass.org/

Re:Internet Issues in the US - Possible Backbone Issue by baldfat on Saturday October 19, 2013 @07:04PM (#6578289)
trace route is nice BUT mtr is better. Mtr does averages on latency at every node and any lost packets.
Re:Internet Issues in the US - Possible Backbone Issue by chimeracoder on Saturday October 19, 2013 @01:28PM (#6577084)
> Hard to do when everything is decentralized.

It's funny, because when you think about it, the Internet is remarkably centralized despite its potential to be almost completely decentralized.

A coworker of mine was telling me just this week that the building we work in, in New York City, is one of two single points of failure for the Northeast (and both points of failure are in New York).

This is an incredibly chilling thought when you consider just how little control humans still have over natural disasters (we saw this firsthand last year in NYC).

By "single point of failure" I don't mean that every single person would be unable to access every single website, but that large chunks of the Internet (particularly anything coming from Europe) would be unavailable for most of those users.

Re:Internet Issues in the US - Possible Backbone Issue by apaprocki on Saturday October 19, 2013 @03:03PM (#6577395)
Well, you either work in 60 Hudson, or ... :)
Re:Internet Issues in the US - Possible Backbone Issue by chimeracoder on Saturday October 19, 2013 @03:44PM (#6577535)
The one that doesn't have 80,000 gallons of fuel and 2000 gallons of diesel fuel stored in the building. :)

http://en.wikipedia.org/wiki/111_Eighth_Avenue

Re:Internet Issues in the US - Possible Backbone Issue by fatjokes on Saturday October 19, 2013 @06:03PM (#6578074)
Hi Googler!
Re:Internet Issues in the US - Possible Backbone Issue by mrgriscom on Saturday October 19, 2013 @02:06PM (#6577220)
wouldn't that traffic eventually get routed via asia?
Re:Internet Issues in the US - Possible Backbone Issue by medecau on Saturday October 19, 2013 @03:12PM (#6577419)
Or Florida.
Re:Internet Issues in the US - Possible Backbone Issue by kissrdotco on Saturday October 19, 2013 @09:04PM (#6578605)
Apparently it's under construction: http://status.theinternet.net/
Re:Internet Issues in the US - Possible Backbone Issue by aroch on Saturday October 19, 2013 @12:05PM (#6576836)
Well, there is lg.level3.net but it's down too!
Re:Internet Issues in the US - Possible Backbone Issue by JohnTHaller on Saturday October 19, 2013 @11:05AM (#6576625)
According to Internet Pulse, the Level 3 issues seem to be solved as of 11:03am. Still showing a slow connection between AT&T to Cogent and AT&T to SBC. http://www.internetpulse.net/

I am still unable to reach any of the above sites/networks via my Time Warner Cable connection, though.

Re:Internet Issues in the US - Possible Backbone Issue by korphol on Saturday October 19, 2013 @02:31PM (#6577297)
Usually the looking glass for each tier 1 provider will give you a good idea I can see http://lookingglass.level3.net/ is down now.

http://www.bgp4.as/looking-glasses

Re:Internet Issues in the US - Possible Backbone Issue by elliottcarlson on Saturday October 19, 2013 @10:39AM (#6576539)
While most of these services have now been rerouted via Above Net etc, some ISPs such as Time Warner (in NY at least) still seem to be routing via Level3 and unable to get to a lot of destinations.
Re:Internet Issues in the US - Possible Backbone Issue by JohnTHaller on Saturday October 19, 2013 @10:55AM (#6576592)
Time Warner always seems slow to re-route. T-Mobile is having the same issue.
Re:Internet Issues in the US - Possible Backbone Issue by apaprocki on Saturday October 19, 2013 @03:16PM (#6577433)
I guess it's a question for NANOG and not HN (if there are any TWC netops here, chime in!), but I'd love to hear a PM why the very expensive routers they buy are not able to automatically re-route to a different peer AS when this happens.
Re:Internet Issues in the US - Possible Backbone Issue by nayefc on Saturday October 19, 2013 @11:24AM (#6576685)
Because Time Warner sucks.
Re:Internet Issues in the US - Possible Backbone Issue by ajpiano on Saturday October 19, 2013 @11:37AM (#6576733)
Confirm. Woke up this morning to limited connectivity, realised I was getting to some sites and not others. While on hold with TWC (where they said there was an 'outage affecting service in your area'), loaded HN to see if, well, HN would load, and this was the #3 story. Thanks HN, GFY TWC!
Re:Internet Issues in the US - Possible Backbone Issue by schandur on Saturday October 19, 2013 @11:57AM (#6576811)
You just described my morning. Opened HN right after this:

  Tracing route to debian.org [128.31.0.51]
  over a maximum of 30 hops:

  1     2 ms     1 ms     1 ms  192.168.1.1
  2    42 ms    28 ms    28 ms  cpe-74-73-128-1.nyc.res.rr.com [74.73.128.1]
  3    14 ms    13 ms    12 ms  gig-0-2-0-3-nycmnya-rtr1.nyc.rr.com [24.29.98.1]
  4    12 ms    15 ms    15 ms  184.152.112.105
  5    17 ms    15 ms    15 ms  ae-3-0.cr0.nyc20.tbone.rr.com [66.109.6.76]
  6    13 ms    12 ms    15 ms  107.14.17.169
  7    15 ms    12 ms    13 ms  xe-4-2-0.edge4.frankfurt1.level3.net [4.68.63.121]
  8     *        *        *     Request timed out.
  9     *        *        *     Request timed out.
 10     *        *        *     Request timed out.
Re:Internet Issues in the US - Possible Backbone Issue by nayefc on Saturday October 19, 2013 @12:57PM (#6576996)
Time Warner asked me to give them the milliseconds of a working website traceroute, instead of the broken link to a non-working website! It's hilarious (and quite sad) how incompetent Time Warner is.
Re:Internet Issues in the US - Possible Backbone Issue by JohnTHaller on Saturday October 19, 2013 @01:18PM (#6577049)
You should be up and running now.
Re:Internet Issues in the US - Possible Backbone Issue by tux on Saturday October 19, 2013 @11:38AM (#6576735)
I thought this is only associated with my routers "DNS Amplified DDoS vulnerability" on Asus RT-N56U router when using Level3 DNS "209.244.0.3 and 209.244.0.4" but after upgrading my firmware the issue continued. So I've changed my DNS to "8.8.8.8 and 8.8.4.4" google dns and everything started working normal for me. Good to know that this is not just me ^_^
Re:Internet Issues in the US - Possible Backbone Issue by pinkmist on Saturday October 19, 2013 @11:40AM (#6576747)
These issues have been around for at least a couple of weeks now. I too have been monitoring Internet Pulse and what I see is not only Level 3 puking on a regular basis but several of the other backbones also. Quite honestly, it's either a gross lack of competence or the feds are beginning to condition the net as they are in the military. ~Ginn
Re:Internet Issues in the US - Possible Backbone Issue by joezydeco on Saturday October 19, 2013 @12:02PM (#6576822)
We lost telecom for almost all of Thursday and our provider told us there was a major metro fiber cut somewhere. Related?
Re:Internet Issues in the US - Possible Backbone Issue by dec0dedab0de on Saturday October 19, 2013 @12:21PM (#6576886)
I have to remember to check HN before driving in to work when a bunch of people call me about outages.
Re:Internet Issues in the US - Possible Backbone Issue by superuser2 on Saturday October 19, 2013 @02:18PM (#6577265)
Isn't there a website that shows a giant table of all the Tier 1 providers and the current ping latency and packet loss between them? I've been unable to find this again... does anyone have a link?
Re:Internet Issues in the US - Possible Backbone Issue by agnokapathetic on Saturday October 19, 2013 @02:26PM (#6577291)
Re:Internet Issues in the US - Possible Backbone Issue by patrickg_zill on Saturday October 19, 2013 @02:57PM (#6577382)
Level3 is in the midst of grooming/updating/changing some of their fiber connections.

One Level3 DC I have servers in, was served out of Baltimore, MD ; but in the last few weeks they changed it to be served out of NYC. By "served" I mean that the actual huge-Cisco that the fiber terminated to, was in Baltimore.

Very glad to have moved out of Level3, actually :)

Re:Internet Issues in the US - Possible Backbone Issue by wglb on Sunday October 20, 2013 @07:40PM (#6582103)
This http://www.internetpulse.net/ is always a handy tool to get an internet-wide picture.
Re:Internet Issues in the US - Possible Backbone Issue by gesman on Saturday October 19, 2013 @12:16PM (#6576875)
My dedicated server went on to serving data as dialup modem speed and provider said it was optic fiber issues on their end. That seemed to be fixed later on.
Re:Internet Issues in the US - Possible Backbone Issue by neals on Saturday October 19, 2013 @10:36AM (#6576531)
what is a level 3?
Re:Internet Issues in the US - Possible Backbone Issue by johnchristopher on Saturday October 19, 2013 @10:38AM (#6576536)
Re:Internet Issues in the US - Possible Backbone Issue by biot on Saturday October 19, 2013 @04:26PM (#6577666)
Answer: See What is Google?
Re:Internet Issues in the US - Possible Backbone Issue by JohnTHaller on Saturday October 19, 2013 @01:10PM (#6577031)
As of 1pm EDT, I can now reach Rackspace, Facebook, Hotmail, Github, Digital Ocean, etc.
Re:Internet Issues in the US - Possible Backbone Issue by TimHordern on Saturday October 19, 2013 @12:25PM (#6576894)
We're beginning to see sites come back up on our TWC connection.
Re:Internet Issues in the US - Possible Backbone Issue by zebra on Saturday October 19, 2013 @10:21AM (#6576490)
Maybe NSA filter overload?
Re:Internet Issues in the US - Possible Backbone Issue by motoboi on Saturday October 19, 2013 @12:45PM (#6576965)
NSA updating firmwares.
Re:Internet Issues in the US - Possible Backbone Issue by JohnTHaller on Saturday October 19, 2013 @10:51AM (#6576579)
11 out of 37 routers down as of 11am EDT. http://www.internettrafficreport.com/namerica.htm
Re:Internet Issues in the US - Possible Backbone Issue by chrisbolt on Saturday October 19, 2013 @03:54PM (#6577566)
Please stop linking to this site. It's been abandoned for ages and isn't a useful indicator of anything.
Re:Internet Issues in the US - Possible Backbone Issue by jessaustin on Saturday October 19, 2013 @10:58AM (#6576603)
Are they typically all up?
Re:Internet Issues in the US - Possible Backbone Issue by JohnTHaller on Saturday October 19, 2013 @01:10PM (#6577029)
Nope. But that seemed like a few more than normal to me. I might be wrong about that, though.