I'm in Switzerland and have 25 hops, which can be broken into:

- 1-7: Hops within my ISP's in-country network (~4ms total latency)

- 8-10: Hops within my ISP's in-Europe network (~28ms total latency)

- 11: London -> New York (~93ms total latency)

- 12: New York -> Los Angeles (~160ms total latency)

- 13: Transfer in LA from my ISP to AARNet (about the same latency)

- 14: LA to somewhere in NSW (guessing Sydney, 305ms total latency)

- 15-25: Routing within AARNet and Unimelb (319ms total latency)

So most of the latency looks to be attributed to large hops across oceans rather than internal switching. Even if you could narrow it down to London -> NY -> LA -> NSW you'd have 277ms.

From my university network in Germany, I seem to get a direct London → Perth link and a total latency of 278ms. I couldn't find any information about a direct fiber between London and Australia, though there is one from London to Singapore, and AARNet seems to have presence in Singapore. My guess would be that there is some switching below the IP layer going on in Singapore as described by dicknuckle in a sibling comment.

     9  cr-fra2-be11.x-win.dfn.de (188.1.144.222)  15.360 ms  15.385 ms dfn.mx1.fra.de.geant.net (62.40.124.217)  15.021 ms
    10  ae7.mx1.ams.nl.geant.net (62.40.98.186)  21.806 ms dfn.mx1.fra.de.geant.net (62.40.124.217)  15.244 ms ae7.mx1.ams.nl.geant.net (62.40.98.186)  21.707 ms
    11  ae9.mx1.lon.uk.geant.net (62.40.98.129)  29.059 ms ae7.mx1.ams.nl.geant.net (62.40.98.186)  21.805 ms ae9.mx1.lon.uk.geant.net (62.40.98.129)  28.933 ms
    12  138.44.226.6 (138.44.226.6)  196.613 ms ae9.mx1.lon.uk.geant.net (62.40.98.129)  29.074 ms  29.156 ms
    13  138.44.226.6 (138.44.226.6)  196.807 ms et-7-3-0.pe1.wmlb.vic.aarnet.net.au (113.197.15.28)  277.463 ms 138.44.226.6 (138.44.226.6)  196.778 ms
    14  138.44.64.73 (138.44.64.73)  277.544 ms et-7-3-0.pe1.wmlb.vic.aarnet.net.au (113.197.15.28)  277.629 ms *

I don't think it's Perth, the address has ".vic." in the name, which implies that the router in question is located in Victoria.

My guess is that there's a private network involved.

If you use AARNet's looking glass [0] it shows that its route to GEANT's London router goes through Singapore.

[0]: https://lg.aarnet.edu.au/cgi-bin/lg

Yeah you're right, "pe1" shows up in many of their hostnames with different states in the name. And their looking glass shows a bunch of hops within Australia, none of which show up when I do a traceroute.

PE means provider edge. CE - customer edge.

AARNet own subsea capacity to Singapore: https://www.aarnet.edu.au/network-and-services/the-network/i...

They also have a fairly extensive network of dark fiber and leased fiber in Australia.

I suspect they might have an onward wavelength service to London.

Starlink has no inter-satellite links, so there will still be a large number of hops. The original promise didn't pan out.

Init7 gets there in 15, they reach AARNet directly from their presence in LAX, however it still takes ~310ms to reach NSW.

Huh, my ISP is Init7 and as I said, takes 25 steps to get there, though steps 18-24 inclusive show as "waiting for reply" in MTR.

Init7's traceroute [0] shows 5 fewer steps to r1lon2.core.init7.net than my traceroute though and appears to route through r1bsl1 (assuming Basel) instead of Frankfurt.

Perhaps you're in/near Basel and so skip straight to London, circumventing my 8 hops around Zurich?

[0]: https://www.as13030.net/traceroute.php

Those numbers seem awfully high. I've just ran a speedtest between Germany and California and got 170ms total latency. Back when rabb.it [0] was alive I could even get reaction scores of 450ms compared to 230ms when running the test on my computer. This is quite impressive when you consider that this means video encoding and decoding must happen in less than 50ms. Conventional video encoding is usually slower than real time.

[0] https://en.wikipedia.org/wiki/Rabb.it (Californian startup) [1] https://humanbenchmark.com/tests/reactiontime

Germany to california for you: 170ms. Switzerland to california for that other guy/gal: 160ms

So… awfully high how?

My latency numbers were cumulative, which is what I meant by "total". The total round trip time to Melbourne is 320ms.

> Because the speed of light in a vacuum is 30% faster than in optical fiber

Non-Australians, don't get too excited. Those satellites are at 340 miles, so that adds 680 miles of latency (3.66 ms) plus two to three Starlink hops, which cancels out some of that "speed in a vacuum."

The way to get almost c on earth is via direct microwave links.

The minimum latency is 3.66 ms, but depending on the angle to the satellites involved it may be adding significantly less than 680 miles to the journey. Aka if it’s 340 miles east and 340 miles up that’s 480 miles to the satellite, adding just 140 not 340 miles.

Outside of HFT, most networks are far from the shortest great circle routes between you and the other end, which further completes the issue.

The round-trip is not 140 miles to a satellite at 340 miles.

Regarding cable placement, in some cases it's circuitous. But sometimes not, expecially in 2020.

You misunderstood. (Ignoring the earth being a sphere.)

A satellite directly overhead means you need to travel to that altitude. However, if it’s not overhead light is traveling the hypotenuse of a right triangle where X is the distance to a point underneath the satellite and Y is the altitude of the satellite. That distance is the square root( X^2 + y^2). From there you need to travel to a different base station.

Assuming an ideal path where the satellite is directly between two locations that are 680 miles apart, that adds up to 2 * ( sqrt(340^2 + 340^2) ) ~= 961.7 miles vs 340 + 340 miles, or an added 281.7 miles not 680. In other words 1.414x the distance rather than simply adding 680.

Clearly the earth is not a flat and your very unlikely to be in that situation, but assuming you can reach several satellites at the same time it is likely one of them will be roughly in the direction you want to go.

> The way to get almost c on earth is via direct microwave links.

But line of sight. Short-haul.

You'd be surprised how far HFT companies run their microwave link networks to get better latencies :)

https://sniperinmahwah.wordpress.com/2014/09/22/hft-in-my-ba...

From one space elevator to the next. Yeah?

FYI: both Canada and the US have had transcontinental RF links. The repeaters slow down the transmission by about 30%.

However those were fairly low-bandwidth.

there's probably double or triple the switches in between. they're not really switches though. more like super fast packet routers. the latency is much less than any normal or carrier grade switch. they don't touch the IP layer at all. (I manage a bunch of them)

one of the things I keep telling non-network-eng people is that looking at a traceroute gives you absolutely no way of knowing what the OSI layer 2 topology is for transport networks, MPLS, switches, DWDM circuits carried as lit services "glassed through" a location, etc. Just the IP network.

Discovering the underlying layer 2 topology of a carrier's network requires inside information and cannot be easily discerned sitting at a computer elsewhere on the internet. You might see two routers that appear to be directly adjacent to each other but it's actually carried as a 10Gbps VLAN across a several-state sized region between two cities many hundreds of km apart, with a lot of intermediate equipment in between.

Do you have any recommendations for where to learn more about network eng for us non-network-engs? My experience is in DevOps (on k8s, mostly) and systems/application programming, if that helps pinpoint any advice you could give. I’m trying to get the bigger picture like network engineers understand, as well as start to understand the layer 2 topology that you described.

The traditional and hard way is to start as a first-tier NOC person for an ISP and work one's way up. Or to start as a field tech for an ISP and show skills worthy of getting promoted (perhaps into a NOC job or a manager-of-field-techs job). That method takes quite a while.

Medium to large sized ISPs have a very large amount of BSD/GPL/Apache/misc licensed software running to support back end monitoring and provisioning systems. They do occasionally hire software developers to customize things for their environments, so it certainly wouldn't hurt to reach out to the noteworthy ones in your region and try giving them your CV.

Read the Cisco CCNA1 certification book. It goes deep on the theory and levels of the OSI model like switching. From there it’s a lot easier to grok.

Yep, start there and then throw GNS3 on a fairly hefty VM and start labbing scenarios that relate to environments that are of interest to you.

You could build a playground internet with some VPN-connected boxes which talk BGP and route 10.0.0.0/8 between them. It's a lot of work, but a friend of mine started such a thing and I have learned _so much_ about networking on layers 2 and up from that.

Start by installing a BGP deamon on two boxes that share a network and see what you can do :)

Nanog.org tries to do some outreach. Might be worth checking out some of their intro sessions on the basics of routing, traceroute, etc.

Plenty of other resources like irc channels, too.

Ditto. Would love to figure out if there's a good way of mapping out layer 1/2 connectivity from a single computer as well. Would be really, really fascinating.

If you want layer 1 (physical location of dark fiber) you have to get the GIS dataset from the carriers. The data sets exist, but they're often protected as proprietary information or under NDA. I have a QGIS setup with a ton of stuff in it.

There are other ways of acquiring layer 1 data which are labor intensive and involve the equivalent of filing FOIAs for construction permits with local city and county agencies, etc.

Big facilities based ISPs that have a lot of fiber out there underground and aerial make extensive use of GIS software. Their construction groups will have their own full time GIS staff positions.

You'll have to follow the cable. Since this forwarding is entirely passive there's no way to know.

You could build a lab with older, yet relevant hardware from eBay, etc.

Actually, you don't want to do that. It costs money and you really should have a good door between you and the equipment or you will go nuts from the noise.

Most of the interesting stuff can be setup using Linux, OpenvSwitch, FRR/ BIRD, network namespaces/ VRF and more. It is all in FOS Software - so more or less zero cost. Of course, you will not learn how to configure a Catalyst Switch that way. For some stuff, there are virtual appliances that you can spin up with KVM/ QEMU but most of the enterprise stuff has to be bought. Again, at that time, you will have a solid understanding of what should be happening and will know what to look for in the documentation. The rest is field experience with firmware bugs, methods how to approach some problems and syntactic sugar of the particular equipment. At least that is my view.

I've seen ISP routers massively increasing latency in conjunction with packet loss -- had a problem with one provider recently where packet loss would increase every evening, and so would the rtt. Suggests large buffers on the congested part. The reverse DNS of the given IP suggested it was a gig-e connection (my side of the hop, which was also into LINX, is 100G)

Of course you can hide a router by not decrementing the TTL as it passes through your network at layer 3, you can hide the IP by not responding with ICMP expired messages

An ICMP could well return on a different path to the direction it was sent, with a path like this

Host -> R1 -> R2 -> R3 // R3 -> R4 -> R5 -> R1 -> Host

Traceroute will only show the outbound route, so you should traceroute from both ends

The latency will also be affected by ICMP generation on the router, which could be delayed, rate limited, dropping, etc.

Doing a quick traceroute to a host of mine in Sydney, from London, shows

  5ms to i-91.ulco-core02.telstraglobal.net 202.40.148.33
  82ms to i-10104.unse-core01.telstraglobal.net 202.84.141.145
  132ms to i-10601.1wlt-core02.telstraglobal.net 202.40.148.106
  277ms to i-10406.sydo-core04.telstraglobal.net 202.84.141.226

sydo will be sydney

To get the exact map I could talk to Telstra (in this case I peer with telstra directly in London)

Or I could look at telstra's map, which ddg helpfully tells me is at

https://www.telstraglobal.com/network-infrastructure-map/

Sadly the map doesn't work very well, more form over function

ddging 1wlt-core02.telstraglobal.net returns this though

https://crowdsupport.telstra.com.au/t5/home-broadband/bad-ro...

Which tells me 1wlt is LA. unse will thus be east coast U.S. I'd have expected routing via Singapore.

There's no way to know which way the traffic is actually going without asking Telstra.

I have 2 ethernet circuits from the UK to Washington DC, to me it looks like two layer 2 1500MTU circuit. Only by talking to the provier can I work out which circuits it actually travels on trans atlanticly. It's supposed to be separate, but latency changed by 2ms a few days ago. Asked them about it, and there was a failure in their network, they rerouted in a few hundered milliseconds (which isn't good as now both diverse circuits run via the same equipment, thus any issues like another 100ms outage will cause an impact)

>I have 2 ethernet circuits from the UK to Washington DC, to me it looks like two layer 2 1500MTU circuit. Only by talking to the provier can I work out which circuits it actually travels on trans atlanticly. It's supposed to be separate, but latency changed by 2ms a few days ago. Asked them about it, and there was a failure in their network, they rerouted in a few hundered milliseconds (which isn't good as now both diverse circuits run via the same equipment, thus any issues like another 100ms outage will cause an impact)

Buying two mpls circuits (assuming they are that) from the same provider is a single point of faliure, your only real redundancy is your handover at each site, if they are separate.

You're better of buying an optical link, then you are also not sharing bandwidth with anyone else. The cost isn't that different in my experience, but might be for an trans-Atlantic circuit. IPSEC over normal internet connections with multiple isp's is a better choice.

Don't get me started on that. However the two circuits are sold as a fully resilient pair (different tails from different DCs arrive into the building in Washington different directions with guaranteed different paths to the DCs, then in contract it's guaranteed neither circuit will run via the same equipment or on the same submarine circuit. In the UK they arrive in two separate cities)

However I know reality and contracts never meet, unfortunately in a large disfunctional organization there are other considerations in circuit procurement than technical requirements.

As for internet circuits, I had two ISPs in NY on two separate paths, which is great. Something went wrong about a month ago, and the routing from one of the ISPs changed, meaning that we were back to a single point of failure who we have no business relationship with

We had a partner that bought a redundant mpls service from their DC to another DC (not their own) and to AWS direct connect via that DC. Their mpls provider was a SPOF, the provider for the second DC was a SPOF, then the direct connect circuits terminated in the same AWS router, another SPOF. All 3 had SPOF's had caused its own significant downtime within a few months.

This was designed by the partner, contract signed by us, the whole solution paid by us, without consulting any network engineer on our side until 1 week before it was supposed to be implemented. We proposed two different solutions that would cost less than 1/20 of the cost their solution was costing us ($20k/month), while providing real redundancy (yet to be implemented, though some SPOFs solved by now). Never enjoyed outages as much as this as we got to explain why their solution was so bad each time.

Choir here, although in our case the connectivity was designed by a third party consulting firm, against the advice of long standing employees with experience.

The third party consulting firm are long gone, as are the people high up in the company who brought them in.

Now there's new people high up who come up with the same problems, but in a slightly different way, arguing how their basket is far better than the previous basket.

Sigh.

Massive latency and increase in packet loss, in correlation with a hop between providers, is usually an overly congested peering or transit session. Particularly if noticed most often during evening peak hours. Some people aren't paying attention to their traffic charts and monitoring systems, and aren't upgrading circuits as needed.

100km = 3 light-ms - which should be noticeable in the relative pings, no?

Yes, I mean it can be obvious when ISPs have proper consistent reverse DNS with hostnames and POP designations. You might see a router in Sacramento that is a neighbor of something in Portland. Obviously there's a lot of stuff in between there but it's all abstracted away at the purely layer-3 IP network level.

You've missed one zero, 1 light-ms = 300km.

In vacuum, yes, in medium it's more like 200 km.

Do these have googleable model numbers, if I wanted to see pictures of them ("oh that's what my packets go through")?

At the small end, a juniper MX204 is a good example of a current-gen router that has enough RAM to take multiple full BGP tables, and has a sufficient number of 10 and 100GbE interfaces.

Juniper MX series. Juniper PTX. Cisco ASR 9000.

That's basically a good portion of the internet.

Look at Junipers PTX line, they are fairly common in carrier grade networks AFAICT.

ISP core networks typically have two components, routers and dwdm/opto equipment. The routers are not really special, just your regular ISP core/edge router, adding roughly the same delay for each hop (invisible mpls hops or not) on the order of microseconds.

Opto equipment adds practically no delay as they just forward the light without looking at it or processing it. There might be redundant paths that it can switch between if there's a fiber cut.

I wonder how many hops it will actually remove. I just tested it, and from my apartment in Mountain View, CA to my VPS in a data center ~20 miles away in Fremont, CA I get 14(!) hops.

> I wonder how many hops it will actually remove.

Hopefully a lot! Fun fact, IPv4 and IPv6 have a maximum network size. Because of the TTL field in v4 (renamed max hops in v6), a path through an IP network can never be more than 255 hops long.

Hops are artificial, as long as you are 100% sure you’re not introducing loops you can just tell routers not to decrease the TTL and they’ll be invisible.

It’s just that if you do that and you do introduce a loop the packets will keep looping and the network will very quickly overflow.

You can tunnel though, so the actual amount of hops could be infinite. Routers can also rewrite the field.

And just like NAT with IPv4, we will use hacks to get around implementing a proper solution for decades (and counting).

With satellite internet taking off, and actual interest in extraterrestrial colonies from e.g. NASA and SpaceX, I think we need a proper space communication protocol. Maybe it can all be solved in L2, but I think we will soon be looking at 255 hops the way we look at the limited address space of v4.

It seems people are working on this: https://en.wikipedia.org/wiki/Interplanetary_Internet

Interesting (but probably not too surprising) to see that DTN research is alive and kicking!

I wrote a short seminar paper on DTN protocols as an undergrad in '09, so thanks for the trip down memory lane!

Thanks for posting this, reading about all the progress made with delay tolerant networking is exciting!

It's all about peering. Once they're in more exchanges, and it becomes cost effective to set up peering with them, more traffic will reach them directly asdifferent providers set up peering to both reduce costs across other links and provide better access.

It all depends on peering, I get 13 hops to some networks in Europe which is some 12.000km away from here (7500 freedom units).

Mentioned above, but it won't. The hops are up and down to boats instead of routers.

Routers and other equipment do not add very much delay unless packets sit around in the buffer, and for that you need congestion.

This brings up thoughts of The Hummingbird Project (a movie)

https://youtu.be/vnXLYr6U3bk

It's a bad movie.

The true story is that a company (spread networks) spent many millions building a low latency fiber path between chicago and nyc. Two guys built a lower latency radio network that connected a number of cell towers together and were the first to market rendering that cable obsolete. Within months other people had radio networks up to try to compete (to give an idea about how competitive that space is).