News for hackers. Stuff that matters.
Re:A transport protocol's view of Starlink by CaliforniaKarl on Friday May 17, 2024 @12:00AM (#40386267)
One comment about the analysis of ping times in the section "Low Earth Orbit Systems". Specifically, the analysis of ping times "within each 15s satellite tracking interval".

Most routers do not put ping-processing in the "fast path". That is, instead of having the ping be processed by an ASIC, the ping gets processed by the router's CPU. And ping-processing is typically a lower-priority task. Because of that, you can't assume that the high variation in latency is because of Starlink.

Re:A transport protocol's view of Starlink by simoncion on Friday May 17, 2024 @12:18AM (#40386341)
> That is, instead of having the ping be processed by an ASIC, the ping gets processed by the router's CPU.

I wonder how true that is anymore, with ICMPv6 processing being a mandatory part of IPv6. I could totally see ICMP processing being a low-priority task, but am far less certain that it would not be done by dedicated hardware these days.

On top of that, I've never, ever, ever noticed the behavior he's observing with either my cable ISP connection, or the terrestrial microwave link provided by my local WISP. I don't have enough data to say that I'd never see that behavior if I happened to ping some router powered by a potato... but I've pinged a whole bunch of systems over the years, and have never seen variation like what he describes.

Re:A transport protocol's view of Starlink by toast0 on Friday May 17, 2024 @02:58AM (#40387194)
> I wonder how true that is anymore, with ICMPv6 processing being a mandatory part of IPv6.

ICMPv6 may be mandatory according to specs, but you can still drop most of it with no ill effects. You probably shouldn't drop needs fragmentation packets, but everyone has adapted to those being dropped sometimes, so...

If you ignore neighbor discovery, you'll have trouble reaching your neighbors, though.

But neighbor discovery is low volume mostly, and can be handled by the cpu, not the asic. Needs frag likely won't be directed at the router ip, the asic can forward them just fine though.

I've certainly seen much more variable ping times for routers than for the hosts behind them. If the router's cpu is less busy, ping times are usually a bit more than a host that's right there, and as the router's cpu gets more busy, the rtt increases or pings get dropped. It's not usually a factor of how busy the links are either; routers tend to have a pretty limited amount of cpu and if they're getting a lot of traffic, it's easy to overload them.

I haven't seen it happen quite in the bands like in the article, but sometimes it varies quite a bit. The results tend to be much more consistent if you can get a rtt from full host and not a router.

Re:A transport protocol's view of Starlink by lxgr on Friday May 17, 2024 @12:23AM (#40386367)
I’ve seen ping latencies of up to 30 seconds (yes, 30 000 milliseconds!) with a certain cable ISP, while at the same time VoIP (RDP over regular Internet IP, not “cable voice” or ISP-provided SIP, which often has its own QoS class) was borderline usable.

Could have been traffic shaping or prioritization (their network was in complete shambles after all), but ICMP was definitely taking some lower priority queue or path.

Re:A transport protocol's view of Starlink by kkielhofner on Friday May 17, 2024 @01:34AM (#40386691)
*RTP
Re:A transport protocol's view of Starlink by simoncion on Sunday May 19, 2024 @10:14PM (#40411481)
I'd expect many fewer folks use RTP than use RDP, so I'd expect that "RDP" was absolutely not a typographical error.
Re:A transport protocol's view of Starlink by lxgr on Thursday May 23, 2024 @07:31PM (#40461161)
I did indeed mean RTP! Almost everybody uses RTP these days, whether they realize it or not. It’s the basis for WebRTC and most proprietary VoIP solutions.
Re:A transport protocol's view of Starlink by jiveturkey on Friday May 17, 2024 @12:35AM (#40386424)
Indeed. Surprising that such a technical article from a supremely technical and knowledgeable person would not at least have a disclaimer about it. I would also be interested to know how much a Starlink satellite is an ASIC router vs pure CPU.

I believe a better technique would be to ping _through_ the router to another endpoint under ones own control, on the same network as the origin client, and where processing latency can be controlled. Basically ping yourself but via starlink and back.

That said, in practical terms he's quite right about the jitter problem. I maintain 2 networks myself at my remote home. One my microwave WISP for tight latency and jitter, and then Starlink for much better bandwidth/throughput.

Re:A transport protocol's view of Starlink by alfons_foobar on Friday May 17, 2024 @01:21AM (#40386635)
He does ping through the starlink terminal:

> ping across a Starlink connection from the customer side terminal to the first IP end point behind the Starlink earth station

Re:A transport protocol's view of Starlink by jiveturkey on Friday May 17, 2024 @02:54PM (#40393112)
I interpreted that statement incorrectly as being an endpoint on the satellite itself. I guess the "earth station" is the round trip terrestrial endpoint via the satellite. I was misreading it as being the local terminal.
Re:A transport protocol's view of Starlink by alfons_foobar on Friday May 17, 2024 @01:20AM (#40386630)
That is true for ICMP packets directed at the router itself.

Pretty sure that is the reason why he used

> ping across a Starlink connection from the customer side terminal to the first IP end point behind the Starlink earth station

instead of pinging the terminal itself.

(ICMP packets passing _through_ a router are handled just like any other traffic and are therefore OK for an analysis like this)

Re:A transport protocol's view of Starlink by maxmouchet on Friday May 17, 2024 @01:37AM (#40386708)
Additionally, I would not expect such “baseline” changes in the minimum RTT to be due to variation in CPU processing time.

These changes are more typical of a physical path change, as suggested by the author.

CPU/soft processing latency would look more like additive noise.

I give some examples of RTT patterns in this talk: https://ripe77.ripe.net/archives/video/2250/

Re:A transport protocol's view of Starlink by CaliforniaKarl on Friday May 17, 2024 @01:45AM (#40386759)
It's unclear to me. `traceroute` typically just returns router IPs, so I'm assuming that "the first IP end point behind the Starlink earth station" is the IP address of a router of some sort.
Re:A transport protocol's view of Starlink by alfons_foobar on Friday May 17, 2024 @02:14AM (#40386958)
Hm, I understood "ip endpoint" to mean something else than a router (because I am very sure that Geoff is aware of the "ICMP is not handled on the ASIC"-issue).
Re:A transport protocol's view of Starlink by chgs on Friday May 17, 2024 @02:19AM (#40386990)
I ping routers around the world all the time and rarely see any significant (>1ms) change
Re:A transport protocol's view of Starlink by damsalor on Friday May 17, 2024 @02:44AM (#40387116)
Most ppl do not enjoy that privilege
Re:A transport protocol's view of Starlink by alfons_foobar on Friday May 17, 2024 @05:38AM (#40388071)
What do you mean by "privilege" in this context?
Re:A transport protocol's view of Starlink by piltdownman on Friday May 17, 2024 @06:38AM (#40388398)
1st World Infrastructure and Routing, being based in a densely populated and developed locale I'd imagine.

YMMV if you attempt the same traceroute from Bangalore or Sydney to London/Paris/NYC.

Re:A transport protocol's view of Starlink by bayindirh on Friday May 17, 2024 @09:16AM (#40389534)
I just tested something to a similar effect. It's much more jumpy time-wise, yes.
Re:A transport protocol's view of Starlink by chgs on Friday May 17, 2024 @02:13PM (#40392686)
Just pinged my isp router in Kathmandu for 30 seconds, 172-174ms return. It’s on AS4007.

From my router in Dhaka pining a router on airtel in India, 36-38ms

Routers in the real would give very stable ping responses, no matter where they are based.

Re:A transport protocol's view of Starlink by inemesitaffia on Monday May 20, 2024 @02:56PM (#40418833)
There was a time latency on AT&T 3G was 9 seconds during congestion.

The last mile is usually the issue. Pretty much only Cogent and HE allow you to have packet loss

Re:A transport protocol's view of Starlink by chgs on Monday May 20, 2024 @03:27PM (#40419208)
The argument was “Most routers do not put ping-processing in the "fast path". That is, instead of having the ping be processed by an ASIC, the ping gets processed by the router's CPU.”

That’s meaningless. Control planes are often policed sure, but on overload will simply drop. In my experience they drop icmp to expired before echo response but most will generate ttl expired just fine.

Any router capable of processing a full bgp table, and to be honest any router made in the last 20 years, is perfectly capable of responding to icmp echos.

There was then a second argument that “3rd world” routers aren’t as good as ones in western country. In the majority of cases they’re exactly the same. That western arrogance is somewhere between insulting and amusing.

The final argument is about path loss/jitter/etc, specially loss on the first hop (your “my crappy 3G provider” argument)

That’s exactly what this test of starlink is showing.

Starlink is a great tool in specific cases, but the fanboyish ness often drowns out the actual benefits.

Re:A transport protocol's view of Starlink by inemesitaffia on Wednesday May 22, 2024 @04:26AM (#40438654)
It's not unusual to see occasionally missing routers in a traceroute, please.

You be capping

Re:A transport protocol's view of Starlink by chgs on Monday May 20, 2024 @03:32PM (#40419261)
About a week ago there were significant routing issues between east Africa and Europe. Latency ballooned, and jitter was quite significant too - especially over thr course of the day. I believe both EASSY and one of the semewe circuits were down. I had packets from the U.K. reaching Mozambique via Tokyo.

That was a temporary situation though, and I think the worst I’ve seen in east Africa from the last 10 years.