Not to mention the dramatic reduction of hops. My route from the USA east coast to the university of Melbourne in Australia[1] is 30 hops reported by traceroute, with at least as many switches in the way. You could make the same link with only a few satellites.

[1]www.ie.unimelb.edu.au

EDIT: 30 is actually just the default max hops in traceroute, its really 32 hops from me to Melbourne.

- 1-7: Hops within my ISP's in-country network (~4ms total latency)

- 8-10: Hops within my ISP's in-Europe network (~28ms total latency)

- 11: London -> New York (~93ms total latency)

- 12: New York -> Los Angeles (~160ms total latency)

- 13: Transfer in LA from my ISP to AARNet (about the same latency)

- 14: LA to somewhere in NSW (guessing Sydney, 305ms total latency)

- 15-25: Routing within AARNet and Unimelb (319ms total latency)

So most of the latency looks to be attributed to large hops across oceans rather than internal switching. Even if you could narrow it down to London -> NY -> LA -> NSW you'd have 277ms.

     9  cr-fra2-be11.x-win.dfn.de (188.1.144.222)  15.360 ms  15.385 ms dfn.mx1.fra.de.geant.net (62.40.124.217)  15.021 ms
    10  ae7.mx1.ams.nl.geant.net (62.40.98.186)  21.806 ms dfn.mx1.fra.de.geant.net (62.40.124.217)  15.244 ms ae7.mx1.ams.nl.geant.net (62.40.98.186)  21.707 ms
    11  ae9.mx1.lon.uk.geant.net (62.40.98.129)  29.059 ms ae7.mx1.ams.nl.geant.net (62.40.98.186)  21.805 ms ae9.mx1.lon.uk.geant.net (62.40.98.129)  28.933 ms
    12  138.44.226.6 (138.44.226.6)  196.613 ms ae9.mx1.lon.uk.geant.net (62.40.98.129)  29.074 ms  29.156 ms
    13  138.44.226.6 (138.44.226.6)  196.807 ms et-7-3-0.pe1.wmlb.vic.aarnet.net.au (113.197.15.28)  277.463 ms 138.44.226.6 (138.44.226.6)  196.778 ms
    14  138.44.64.73 (138.44.64.73)  277.544 ms et-7-3-0.pe1.wmlb.vic.aarnet.net.au (113.197.15.28)  277.629 ms *

My guess is that there's a private network involved.

If you use AARNet's looking glass [0] it shows that its route to GEANT's London router goes through Singapore.

[0]: https://lg.aarnet.edu.au/cgi-bin/lg

They also have a fairly extensive network of dark fiber and leased fiber in Australia.

I suspect they might have an onward wavelength service to London.

Init7's traceroute [0] shows 5 fewer steps to r1lon2.core.init7.net than my traceroute though and appears to route through r1bsl1 (assuming Basel) instead of Frankfurt.

Perhaps you're in/near Basel and so skip straight to London, circumventing my 8 hops around Zurich?

[0]: https://www.as13030.net/traceroute.php

[0] https://en.wikipedia.org/wiki/Rabb.it (Californian startup) [1] https://humanbenchmark.com/tests/reactiontime

So… awfully high how?

Non-Australians, don't get too excited. Those satellites are at 340 miles, so that adds 680 miles of latency (3.66 ms) plus two to three Starlink hops, which cancels out some of that "speed in a vacuum."

The way to get almost c on earth is via direct microwave links.

Outside of HFT, most networks are far from the shortest great circle routes between you and the other end, which further completes the issue.

Regarding cable placement, in some cases it's circuitous. But sometimes not, expecially in 2020.

A satellite directly overhead means you need to travel to that altitude. However, if it’s not overhead light is traveling the hypotenuse of a right triangle where X is the distance to a point underneath the satellite and Y is the altitude of the satellite. That distance is the square root( X^2 + y^2). From there you need to travel to a different base station.

Assuming an ideal path where the satellite is directly between two locations that are 680 miles apart, that adds up to 2 * ( sqrt(340^2 + 340^2) ) ~= 961.7 miles vs 340 + 340 miles, or an added 281.7 miles not 680. In other words 1.414x the distance rather than simply adding 680.

Clearly the earth is not a flat and your very unlikely to be in that situation, but assuming you can reach several satellites at the same time it is likely one of them will be roughly in the direction you want to go.

But line of sight. Short-haul.

https://sniperinmahwah.wordpress.com/2014/09/22/hft-in-my-ba...

However those were fairly low-bandwidth.

Discovering the underlying layer 2 topology of a carrier's network requires inside information and cannot be easily discerned sitting at a computer elsewhere on the internet. You might see two routers that appear to be directly adjacent to each other but it's actually carried as a 10Gbps VLAN across a several-state sized region between two cities many hundreds of km apart, with a lot of intermediate equipment in between.

Medium to large sized ISPs have a very large amount of BSD/GPL/Apache/misc licensed software running to support back end monitoring and provisioning systems. They do occasionally hire software developers to customize things for their environments, so it certainly wouldn't hurt to reach out to the noteworthy ones in your region and try giving them your CV.

Start by installing a BGP deamon on two boxes that share a network and see what you can do :)

Plenty of other resources like irc channels, too.

There are other ways of acquiring layer 1 data which are labor intensive and involve the equivalent of filing FOIAs for construction permits with local city and county agencies, etc.

Big facilities based ISPs that have a lot of fiber out there underground and aerial make extensive use of GIS software. Their construction groups will have their own full time GIS staff positions.

Most of the interesting stuff can be setup using Linux, OpenvSwitch, FRR/ BIRD, network namespaces/ VRF and more. It is all in FOS Software - so more or less zero cost. Of course, you will not learn how to configure a Catalyst Switch that way. For some stuff, there are virtual appliances that you can spin up with KVM/ QEMU but most of the enterprise stuff has to be bought. Again, at that time, you will have a solid understanding of what should be happening and will know what to look for in the documentation. The rest is field experience with firmware bugs, methods how to approach some problems and syntactic sugar of the particular equipment. At least that is my view.

Of course you can hide a router by not decrementing the TTL as it passes through your network at layer 3, you can hide the IP by not responding with ICMP expired messages

An ICMP could well return on a different path to the direction it was sent, with a path like this

Host -> R1 -> R2 -> R3 // R3 -> R4 -> R5 -> R1 -> Host

Traceroute will only show the outbound route, so you should traceroute from both ends

The latency will also be affected by ICMP generation on the router, which could be delayed, rate limited, dropping, etc.

Doing a quick traceroute to a host of mine in Sydney, from London, shows

  5ms to i-91.ulco-core02.telstraglobal.net 202.40.148.33
  82ms to i-10104.unse-core01.telstraglobal.net 202.84.141.145
  132ms to i-10601.1wlt-core02.telstraglobal.net 202.40.148.106
  277ms to i-10406.sydo-core04.telstraglobal.net 202.84.141.226

To get the exact map I could talk to Telstra (in this case I peer with telstra directly in London)

Or I could look at telstra's map, which ddg helpfully tells me is at

https://www.telstraglobal.com/network-infrastructure-map/

Sadly the map doesn't work very well, more form over function

ddging 1wlt-core02.telstraglobal.net returns this though

https://crowdsupport.telstra.com.au/t5/home-broadband/bad-ro...

Which tells me 1wlt is LA. unse will thus be east coast U.S. I'd have expected routing via Singapore.

There's no way to know which way the traffic is actually going without asking Telstra.

I have 2 ethernet circuits from the UK to Washington DC, to me it looks like two layer 2 1500MTU circuit. Only by talking to the provier can I work out which circuits it actually travels on trans atlanticly. It's supposed to be separate, but latency changed by 2ms a few days ago. Asked them about it, and there was a failure in their network, they rerouted in a few hundered milliseconds (which isn't good as now both diverse circuits run via the same equipment, thus any issues like another 100ms outage will cause an impact)

Buying two mpls circuits (assuming they are that) from the same provider is a single point of faliure, your only real redundancy is your handover at each site, if they are separate.

You're better of buying an optical link, then you are also not sharing bandwidth with anyone else. The cost isn't that different in my experience, but might be for an trans-Atlantic circuit. IPSEC over normal internet connections with multiple isp's is a better choice.

However I know reality and contracts never meet, unfortunately in a large disfunctional organization there are other considerations in circuit procurement than technical requirements.

As for internet circuits, I had two ISPs in NY on two separate paths, which is great. Something went wrong about a month ago, and the routing from one of the ISPs changed, meaning that we were back to a single point of failure who we have no business relationship with

This was designed by the partner, contract signed by us, the whole solution paid by us, without consulting any network engineer on our side until 1 week before it was supposed to be implemented. We proposed two different solutions that would cost less than 1/20 of the cost their solution was costing us ($20k/month), while providing real redundancy (yet to be implemented, though some SPOFs solved by now). Never enjoyed outages as much as this as we got to explain why their solution was so bad each time.

The third party consulting firm are long gone, as are the people high up in the company who brought them in.

Now there's new people high up who come up with the same problems, but in a slightly different way, arguing how their basket is far better than the previous basket.

Sigh.

That's basically a good portion of the internet.

Opto equipment adds practically no delay as they just forward the light without looking at it or processing it. There might be redundant paths that it can switch between if there's a fiber cut.

Hopefully a lot! Fun fact, IPv4 and IPv6 have a maximum network size. Because of the TTL field in v4 (renamed max hops in v6), a path through an IP network can never be more than 255 hops long.

It’s just that if you do that and you do introduce a loop the packets will keep looping and the network will very quickly overflow.

With satellite internet taking off, and actual interest in extraterrestrial colonies from e.g. NASA and SpaceX, I think we need a proper space communication protocol. Maybe it can all be solved in L2, but I think we will soon be looking at 255 hops the way we look at the limited address space of v4.

I wrote a short seminar paper on DTN protocols as an undergrad in '09, so thanks for the trip down memory lane!

https://youtu.be/vnXLYr6U3bk

The true story is that a company (spread networks) spent many millions building a low latency fiber path between chicago and nyc. Two guys built a lower latency radio network that connected a number of cell towers together and were the first to market rendering that cable obsolete. Within months other people had radio networks up to try to compete (to give an idea about how competitive that space is).

One thing to add is that Starship should enable 400 satellites per launch:

https://www.teslarati.com/spacex-president-teases-starship-s...

Put a 100 tonne tungsten rod into space (cost ~$10m), or more likely some cheaper metal. Maybe 10m long with 25cm radius. Stick a cold gas reaction control system, maybe some gridfins and a guidance package. Put it in orbit, Put some ablative on the leading edge. Once in orbit the cold gas system slows the weapon to put it on target. The gridfins and cold gas thrusters control the guidance. The gridfins wouldn't work for long before the burnt off, but recessed cold gas thrusters could continue to work. Ionisation and intense heat would stop any optical sensors or outside guidance, so aiming it would be problematic.

A kinetic warhead with a mass of 100tonnes going at sub orbital speeds (say 7000m/s) would have 2.45x10^12 joules, which is around 0.58kt of tnt. If I remember correctly once it is going past a critical speed the penetrator and target act like fluids and the rod would penetrate a depth based on the relative density of the two materials time the length. Tungsten is about 6 to 12 times as dense as rock, so it might go through 60 to 120m meters of rock.

Non-nuclear, pretty hard to detect or stop. Competitive in price with cruise missiles. Accuracy could be a problem.

Using starship to launch rods that immediately come down on people instead of loitering in space sounds like a much safer use.

I call the opposite, it is extremely easy to detect and track their orbits, a single personal computer can probably track them all and calculate the potential strike point over points in orbits.

Any attempt to deviate from an existing low earth orbit is extremely energy intensive. It also can't strike anywhere - it can be more than 24 hours before its in position over the location it wants to be.

The orbital period for low Earth orbit is closer to 1 hour than 24, and you further reduce that to only a dozen or so minutes by spacing out rods over the orbit... Much like starlink satellites.

Keep in mind an orbit does not cover all of the Earth surface. And unless the target is on the equator, there is no low earth orbit that can maintain it's path over a target consistently.

But for detection, Is that actually easy? I don't know much about it, would it be done optically, or is there a better way? Are there concealment strategies? How about say painting the satellite black?

Fun to think about but I also hope they don't build these things.

A point of clarification, submarines are the only nuclear platform with "undisclosed locations." It is trivial for a nation-state to observe the construction and fitment of a fixed ICBM placement, or track the movements of a truck/rail mounted launcher.

I'm sure U.S. at least tries to keep track of Russian mobile launchers, but to which degree it is successful we wouldn't know for sure. And pretty certain Russia would have problems live tracking launch vehicles (if the USA had any).

50% faster than light in fiber, fiber is 30% slower.

https://youtu.be/m05abdGSOxY

PS: Thanks for the summary and NO Starlink phone. I have gotten sick and tired of people jumping to conclusion suggesting Starlink taking over ISP and Mobile Network. And that is even on HN.

Are you saying 700mbps is slow? Or is that all the bandwidth that the single link can provide (meaning 700 divided by the number of subscribers)? Living in a fly over state, I was lucky at first to get much more than 10mbps. Spectrum has now provided service to my area and I can get 400mbps and am super happy.

https://www.youtube.com/watch?v=y4xBFHjkUvw&feature=youtu.be...

I thought they were redesigning the satellites to be nearly invisible from the ground so as not to disrupt astronomy? Or is that later launches?

Isn't this the 8th? Jesse the Engineer said it was, too. The stage 1 is also on its 5th mission, so rad!

Edit: stage 1 recovery success, and that was perhaps the clearest image of it landing I have ever seen and I've been watching since 2015! Also, I miss the shots of LOX images in zero G they used to do back then.

There was also a full 60 satellite stack of v0.9 satellites that aren't counted towards the "production" constellation.

Now I remember this launch last year around this time, and some of them were not able to maintain their orbit, but weren't the majority of them operational?

So if the Starlink launches happen twice a month, the carbon footprint from the fuel would be at least an order of magnitude smaller than that of a single long haul route of an airline.

Building the satellites and the rocket (including the expendable second stage) is obviously pretty carbon intensive too, but fuel is probably what you were thinking about when posing this question.

https://www.youtube.com/watch?v=C4VHfmiwuv4

https://static.berkeleyearth.org/memos/fugitive-methane-and-...

https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/201200...

TL;DNR: If you take methane out of the atmosphere and you put it back in the atmosphere, the carbon footprint is neutral.

Considering the pay-over-time model that SolarCity implemented, should we not expect the same here? Assuming the user antenna can be removed, it can be rented, just like a modem or router.

Relying on the antennas having a long ‘useful life’, it may not be too prohibitive if rented out.

People who live in a really remote plate and sign up for a 24 month term for some barely-usable VSAT service are usually disappointed to find out how firmly they're locked into the contract, when somebody builds a WISP in their area.

I would not be surprised if there's a terminal rental charge or 12/24/36 month contract terms offered.

Considering the pay-over-time model essentially bankrupted SolarCity, forcing Tesla to buy them out, for which Tesla are being sued by shareholders...I'm not sure that's the model to resurrect.

side note: not related but the Eccentric Orbits book about Iridium was really good imo.

Or if it was scale/can't see it.

Which I suppose even so, that ship would have to be unrealistically massive to have problems flying through one of the gaps.

[1] https://cdn.geekwire.com/wp-content/uploads/2019/02/190208-s...

[1] https://youtu.be/m05abdGSOxY

One of Starlink's competitors OneWeb was able to aquire a user antenna which is apparently a breakthrough in cost reduction at $15 [0]. I would assume the manufacturer has a contractual agreement with OneWeb, but I can imagine that Starlink could develop a similarly priced antenna with it's greater resources.

[0] https://spacenews.com/wyler-claims-breakthrough-in-low-cost-...

Greg Wyler, the founder of Oneweb, has a history of making bold claims that are not really supported by facts.

I'll be surprised if this doesn't shrink. I worked for a company in the mid-2000s that had some pretty cool IP which effectively shrank a briefcase-sized BGAN terminal down to a Pocket PC (pre iPhone days!) with Pocket PC-sized antenna strapped on the back.

https://www.businessinsider.com/elon-musk-spacex-starlink-sa...

The satellites were never the challenging part; cross-links and the user terminal were always the most difficult pieces.

Try to ask this in a non-hyperbolic way, I dont want to sound alarmist and I'm no "chem trail" loon. but "Burning up" doesn't just make it magically go away.

I haven't seen articles on this, so either its being ignored or it's not an issue. :-/

I hope its the later.

[1] https://www.nasa.gov/mission_pages/asteroids/overview/fastfa...

I wonder what the stats of Uranium/Plutonium we let "burn up" in the biosphere is compared to stuff we get while cruising through space is...

[1] https://periodictable.com/Properties/A/MeteoriteAbundance.v.... [2]https://link.springer.com/chapter/10.1007%2F978-1-4419-8694-...