one of the things I keep telling non-network-eng people is that looking at a traceroute gives you absolutely no way of knowing what the OSI layer 2 topology is for transport networks, MPLS, switches, DWDM circuits carried as lit services "glassed through" a location, etc. Just the IP network.

Discovering the underlying layer 2 topology of a carrier's network requires inside information and cannot be easily discerned sitting at a computer elsewhere on the internet. You might see two routers that appear to be directly adjacent to each other but it's actually carried as a 10Gbps VLAN across a several-state sized region between two cities many hundreds of km apart, with a lot of intermediate equipment in between.

Do you have any recommendations for where to learn more about network eng for us non-network-engs? My experience is in DevOps (on k8s, mostly) and systems/application programming, if that helps pinpoint any advice you could give. I’m trying to get the bigger picture like network engineers understand, as well as start to understand the layer 2 topology that you described.

The traditional and hard way is to start as a first-tier NOC person for an ISP and work one's way up. Or to start as a field tech for an ISP and show skills worthy of getting promoted (perhaps into a NOC job or a manager-of-field-techs job). That method takes quite a while.

Medium to large sized ISPs have a very large amount of BSD/GPL/Apache/misc licensed software running to support back end monitoring and provisioning systems. They do occasionally hire software developers to customize things for their environments, so it certainly wouldn't hurt to reach out to the noteworthy ones in your region and try giving them your CV.

Read the Cisco CCNA1 certification book. It goes deep on the theory and levels of the OSI model like switching. From there it’s a lot easier to grok.

Yep, start there and then throw GNS3 on a fairly hefty VM and start labbing scenarios that relate to environments that are of interest to you.

You could build a playground internet with some VPN-connected boxes which talk BGP and route 10.0.0.0/8 between them. It's a lot of work, but a friend of mine started such a thing and I have learned _so much_ about networking on layers 2 and up from that.

Start by installing a BGP deamon on two boxes that share a network and see what you can do :)

Nanog.org tries to do some outreach. Might be worth checking out some of their intro sessions on the basics of routing, traceroute, etc.

Plenty of other resources like irc channels, too.

Ditto. Would love to figure out if there's a good way of mapping out layer 1/2 connectivity from a single computer as well. Would be really, really fascinating.

If you want layer 1 (physical location of dark fiber) you have to get the GIS dataset from the carriers. The data sets exist, but they're often protected as proprietary information or under NDA. I have a QGIS setup with a ton of stuff in it.

There are other ways of acquiring layer 1 data which are labor intensive and involve the equivalent of filing FOIAs for construction permits with local city and county agencies, etc.

Big facilities based ISPs that have a lot of fiber out there underground and aerial make extensive use of GIS software. Their construction groups will have their own full time GIS staff positions.

You'll have to follow the cable. Since this forwarding is entirely passive there's no way to know.

You could build a lab with older, yet relevant hardware from eBay, etc.

Actually, you don't want to do that. It costs money and you really should have a good door between you and the equipment or you will go nuts from the noise.

Most of the interesting stuff can be setup using Linux, OpenvSwitch, FRR/ BIRD, network namespaces/ VRF and more. It is all in FOS Software - so more or less zero cost. Of course, you will not learn how to configure a Catalyst Switch that way. For some stuff, there are virtual appliances that you can spin up with KVM/ QEMU but most of the enterprise stuff has to be bought. Again, at that time, you will have a solid understanding of what should be happening and will know what to look for in the documentation. The rest is field experience with firmware bugs, methods how to approach some problems and syntactic sugar of the particular equipment. At least that is my view.

I've seen ISP routers massively increasing latency in conjunction with packet loss -- had a problem with one provider recently where packet loss would increase every evening, and so would the rtt. Suggests large buffers on the congested part. The reverse DNS of the given IP suggested it was a gig-e connection (my side of the hop, which was also into LINX, is 100G)

Of course you can hide a router by not decrementing the TTL as it passes through your network at layer 3, you can hide the IP by not responding with ICMP expired messages

An ICMP could well return on a different path to the direction it was sent, with a path like this

Host -> R1 -> R2 -> R3 // R3 -> R4 -> R5 -> R1 -> Host

Traceroute will only show the outbound route, so you should traceroute from both ends

The latency will also be affected by ICMP generation on the router, which could be delayed, rate limited, dropping, etc.

Doing a quick traceroute to a host of mine in Sydney, from London, shows

  5ms to i-91.ulco-core02.telstraglobal.net 202.40.148.33
  82ms to i-10104.unse-core01.telstraglobal.net 202.84.141.145
  132ms to i-10601.1wlt-core02.telstraglobal.net 202.40.148.106
  277ms to i-10406.sydo-core04.telstraglobal.net 202.84.141.226

sydo will be sydney

To get the exact map I could talk to Telstra (in this case I peer with telstra directly in London)

Or I could look at telstra's map, which ddg helpfully tells me is at

https://www.telstraglobal.com/network-infrastructure-map/

Sadly the map doesn't work very well, more form over function

ddging 1wlt-core02.telstraglobal.net returns this though

https://crowdsupport.telstra.com.au/t5/home-broadband/bad-ro...

Which tells me 1wlt is LA. unse will thus be east coast U.S. I'd have expected routing via Singapore.

There's no way to know which way the traffic is actually going without asking Telstra.

I have 2 ethernet circuits from the UK to Washington DC, to me it looks like two layer 2 1500MTU circuit. Only by talking to the provier can I work out which circuits it actually travels on trans atlanticly. It's supposed to be separate, but latency changed by 2ms a few days ago. Asked them about it, and there was a failure in their network, they rerouted in a few hundered milliseconds (which isn't good as now both diverse circuits run via the same equipment, thus any issues like another 100ms outage will cause an impact)

>I have 2 ethernet circuits from the UK to Washington DC, to me it looks like two layer 2 1500MTU circuit. Only by talking to the provier can I work out which circuits it actually travels on trans atlanticly. It's supposed to be separate, but latency changed by 2ms a few days ago. Asked them about it, and there was a failure in their network, they rerouted in a few hundered milliseconds (which isn't good as now both diverse circuits run via the same equipment, thus any issues like another 100ms outage will cause an impact)

Buying two mpls circuits (assuming they are that) from the same provider is a single point of faliure, your only real redundancy is your handover at each site, if they are separate.

You're better of buying an optical link, then you are also not sharing bandwidth with anyone else. The cost isn't that different in my experience, but might be for an trans-Atlantic circuit. IPSEC over normal internet connections with multiple isp's is a better choice.

Don't get me started on that. However the two circuits are sold as a fully resilient pair (different tails from different DCs arrive into the building in Washington different directions with guaranteed different paths to the DCs, then in contract it's guaranteed neither circuit will run via the same equipment or on the same submarine circuit. In the UK they arrive in two separate cities)

However I know reality and contracts never meet, unfortunately in a large disfunctional organization there are other considerations in circuit procurement than technical requirements.

As for internet circuits, I had two ISPs in NY on two separate paths, which is great. Something went wrong about a month ago, and the routing from one of the ISPs changed, meaning that we were back to a single point of failure who we have no business relationship with

We had a partner that bought a redundant mpls service from their DC to another DC (not their own) and to AWS direct connect via that DC. Their mpls provider was a SPOF, the provider for the second DC was a SPOF, then the direct connect circuits terminated in the same AWS router, another SPOF. All 3 had SPOF's had caused its own significant downtime within a few months.

This was designed by the partner, contract signed by us, the whole solution paid by us, without consulting any network engineer on our side until 1 week before it was supposed to be implemented. We proposed two different solutions that would cost less than 1/20 of the cost their solution was costing us ($20k/month), while providing real redundancy (yet to be implemented, though some SPOFs solved by now). Never enjoyed outages as much as this as we got to explain why their solution was so bad each time.

Choir here, although in our case the connectivity was designed by a third party consulting firm, against the advice of long standing employees with experience.

The third party consulting firm are long gone, as are the people high up in the company who brought them in.

Now there's new people high up who come up with the same problems, but in a slightly different way, arguing how their basket is far better than the previous basket.

Sigh.

Massive latency and increase in packet loss, in correlation with a hop between providers, is usually an overly congested peering or transit session. Particularly if noticed most often during evening peak hours. Some people aren't paying attention to their traffic charts and monitoring systems, and aren't upgrading circuits as needed.

100km = 3 light-ms - which should be noticeable in the relative pings, no?

Yes, I mean it can be obvious when ISPs have proper consistent reverse DNS with hostnames and POP designations. You might see a router in Sacramento that is a neighbor of something in Portland. Obviously there's a lot of stuff in between there but it's all abstracted away at the purely layer-3 IP network level.

You've missed one zero, 1 light-ms = 300km.

In vacuum, yes, in medium it's more like 200 km.

Do these have googleable model numbers, if I wanted to see pictures of them ("oh that's what my packets go through")?

At the small end, a juniper MX204 is a good example of a current-gen router that has enough RAM to take multiple full BGP tables, and has a sufficient number of 10 and 100GbE interfaces.

Juniper MX series. Juniper PTX. Cisco ASR 9000.

That's basically a good portion of the internet.

Look at Junipers PTX line, they are fairly common in carrier grade networks AFAICT.

ISP core networks typically have two components, routers and dwdm/opto equipment. The routers are not really special, just your regular ISP core/edge router, adding roughly the same delay for each hop (invisible mpls hops or not) on the order of microseconds.

Opto equipment adds practically no delay as they just forward the light without looking at it or processing it. There might be redundant paths that it can switch between if there's a fiber cut.