News for hackers. Stuff that matters.
Re:SpaceX Starlink is now its own ASN and present at the Sea... by iso1631 on Thursday June 04, 2020 @06:27AM (#23414800)
I've seen ISP routers massively increasing latency in conjunction with packet loss -- had a problem with one provider recently where packet loss would increase every evening, and so would the rtt. Suggests large buffers on the congested part. The reverse DNS of the given IP suggested it was a gig-e connection (my side of the hop, which was also into LINX, is 100G)

Of course you can hide a router by not decrementing the TTL as it passes through your network at layer 3, you can hide the IP by not responding with ICMP expired messages

An ICMP could well return on a different path to the direction it was sent, with a path like this

Host -> R1 -> R2 -> R3 // R3 -> R4 -> R5 -> R1 -> Host

Traceroute will only show the outbound route, so you should traceroute from both ends

The latency will also be affected by ICMP generation on the router, which could be delayed, rate limited, dropping, etc.

Doing a quick traceroute to a host of mine in Sydney, from London, shows

  5ms to i-91.ulco-core02.telstraglobal.net 202.40.148.33
  82ms to i-10104.unse-core01.telstraglobal.net 202.84.141.145
  132ms to i-10601.1wlt-core02.telstraglobal.net 202.40.148.106
  277ms to i-10406.sydo-core04.telstraglobal.net 202.84.141.226
sydo will be sydney

To get the exact map I could talk to Telstra (in this case I peer with telstra directly in London)

Or I could look at telstra's map, which ddg helpfully tells me is at

https://www.telstraglobal.com/network-infrastructure-map/

Sadly the map doesn't work very well, more form over function

ddging 1wlt-core02.telstraglobal.net returns this though

https://crowdsupport.telstra.com.au/t5/home-broadband/bad-ro...

Which tells me 1wlt is LA. unse will thus be east coast U.S. I'd have expected routing via Singapore.

There's no way to know which way the traffic is actually going without asking Telstra.

I have 2 ethernet circuits from the UK to Washington DC, to me it looks like two layer 2 1500MTU circuit. Only by talking to the provier can I work out which circuits it actually travels on trans atlanticly. It's supposed to be separate, but latency changed by 2ms a few days ago. Asked them about it, and there was a failure in their network, they rerouted in a few hundered milliseconds (which isn't good as now both diverse circuits run via the same equipment, thus any issues like another 100ms outage will cause an impact)

Re:SpaceX Starlink is now its own ASN and present at the Sea... by Hikikomori on Thursday June 04, 2020 @07:01AM (#23415011)
>I have 2 ethernet circuits from the UK to Washington DC, to me it looks like two layer 2 1500MTU circuit. Only by talking to the provier can I work out which circuits it actually travels on trans atlanticly. It's supposed to be separate, but latency changed by 2ms a few days ago. Asked them about it, and there was a failure in their network, they rerouted in a few hundered milliseconds (which isn't good as now both diverse circuits run via the same equipment, thus any issues like another 100ms outage will cause an impact)

Buying two mpls circuits (assuming they are that) from the same provider is a single point of faliure, your only real redundancy is your handover at each site, if they are separate.

You're better of buying an optical link, then you are also not sharing bandwidth with anyone else. The cost isn't that different in my experience, but might be for an trans-Atlantic circuit. IPSEC over normal internet connections with multiple isp's is a better choice.

Re:SpaceX Starlink is now its own ASN and present at the Sea... by iso1631 on Thursday June 04, 2020 @07:41AM (#23415317)
Don't get me started on that. However the two circuits are sold as a fully resilient pair (different tails from different DCs arrive into the building in Washington different directions with guaranteed different paths to the DCs, then in contract it's guaranteed neither circuit will run via the same equipment or on the same submarine circuit. In the UK they arrive in two separate cities)

However I know reality and contracts never meet, unfortunately in a large disfunctional organization there are other considerations in circuit procurement than technical requirements.

As for internet circuits, I had two ISPs in NY on two separate paths, which is great. Something went wrong about a month ago, and the routing from one of the ISPs changed, meaning that we were back to a single point of failure who we have no business relationship with

Re:SpaceX Starlink is now its own ASN and present at the Sea... by Hikikomori on Thursday June 04, 2020 @08:29AM (#23415705)
We had a partner that bought a redundant mpls service from their DC to another DC (not their own) and to AWS direct connect via that DC. Their mpls provider was a SPOF, the provider for the second DC was a SPOF, then the direct connect circuits terminated in the same AWS router, another SPOF. All 3 had SPOF's had caused its own significant downtime within a few months.

This was designed by the partner, contract signed by us, the whole solution paid by us, without consulting any network engineer on our side until 1 week before it was supposed to be implemented. We proposed two different solutions that would cost less than 1/20 of the cost their solution was costing us ($20k/month), while providing real redundancy (yet to be implemented, though some SPOFs solved by now). Never enjoyed outages as much as this as we got to explain why their solution was so bad each time.

Re:SpaceX Starlink is now its own ASN and present at the Sea... by iso1631 on Thursday June 04, 2020 @02:20PM (#23419944)
Choir here, although in our case the connectivity was designed by a third party consulting firm, against the advice of long standing employees with experience.

The third party consulting firm are long gone, as are the people high up in the company who brought them in.

Now there's new people high up who come up with the same problems, but in a slightly different way, arguing how their basket is far better than the previous basket.

Sigh.

Re:SpaceX Starlink is now its own ASN and present at the Sea... by walrus01 on Thursday June 04, 2020 @02:36PM (#23420133)
Massive latency and increase in packet loss, in correlation with a hop between providers, is usually an overly congested peering or transit session. Particularly if noticed most often during evening peak hours. Some people aren't paying attention to their traffic charts and monitoring systems, and aren't upgrading circuits as needed.