Irrespective of how right or wrong your first paragraph is; this is one of the reasons I absolutely love the HN community.
If you saw a lot of jitter it was because you were in an oversubscribed, contended tdma network. What was your terminal hardware, what modems were you using and what was the rf chain and modem setup at the far end (not in Afghanistan)?
Latency on terrestrial ptp microwave also does not vary. Ducting and diffraction and temperature inversions on long 6GHz links affects rsl and modulation, not latency.
In the case of starlink, each satellite is functioning as a bent pipe relay between cpe and starlink earth station. Latency will vary a ms or two based on satellite position during a pass overhead, but no more than that. The rest of the latency is the same as from any terrestrial internet customer in the same geographic location as the earth station. For instance one of the beta test earth stations is in north bend, wa, colocated with a level3 long haul dwdm site.
People in WA have been posting tests with latencies of 25-26ms to speed test servers in downtown Seattle. That's actually really, really good, considering that latency to your default gateway on a residential cable docsis3 coaxial connection can easily be 16ms on a terrestrial connection. North bend is a few ms from seattle, the brewster starlink teleport site several ms more.
In the case of starlink they are competing against 32:1 or worse oversubscribed geostationary consumer grade vsat services like viasat and Hughesnet. Starlink is far superior tech.
Lost packets increase latency. And weather conditions cause lost packets.
As a note there were lots of people buying and installing cheap COTS 1.2-1.8 meter size ku band vsat terminals at all sorts of military sites in Afghanistan for MWR purposes, it's not rocket science or really hard to do at all. This stuff was, and is, all kept air gapped and separated from anything that is a real DoD or ISAF/RS network. These were networks for peoples' own personal recreational internet and entertainment in their rest time. You absolutely would see weird latency, jitter, packet loss behavior on those.
They can both be right as walrus01 seems to talk about latency of a single rtt in isolation, and withinboredom about what does the TCP endpoint effectively see.
Example: at one location, a 3.8 meter ku band dual linear polarity compact cassegrain dish in somalia, a 200W BUC+SSPA, good quality norsat PLL LNB, a cdm760 modem. Ten MHz of transponder space occupied total. Other end, a 5.5m dish with similar gear on it in a teleport in Cyprus.
If the person I'm replying to isn't willing or able to clarify whether they were in a contended, or dedicated network, or specify the gear, it's hard to say exactly why they're seeing jitter. But I can quite confidently say that excursions above 495ms are not something seen in a dedicated network.
It's how that dedicated capacity can get sliced up and shared that is the most common cause of packet loss, jitter and high latency in geostationary based two way satellite IP links.
The most common thing for a cheap VSAT isp to do is to acquire one chunk of Ku transponder capacity (let's say about $8000 a month worth), install something like an idirect hub with line cards in it, and sell access to individual end users in $developingnation at an 8:1 oversubscription ratio or more. Meaning that an end user vsat terminal might see 495ms at a lucky time when its peers in the same network were not also moving traffic, but could also often see latency anywhere from 500 to 850 during busy periods.
Now that same 8000 bucks a month of transponder space could equally well be assigned to a single high priority end user with money to pay, in which case they would never see above 495ms. Or it could be oversubscribed to all hell, such as if the ISP wanted to sell $280/month service to a bunch of very cost conscious end users in a developing nation environment who can only afford very cheap VSAT terminal hardware (one 1.2m fiberglass compression molded reflector, a 4W BUC, and an idirect modem) and minimal monthly ongoing payments. In which case each user will be in a 16:1 or worse oversubscribed network.
The economics of making a 5500 kilogram $200 million satellite in geostationary orbit pay for itself, before its lifespan is done in 15 years, are not favorable. If you ever want to truly be shocked go price the $ per Mbps per month that results from the 1:1 dedicated setup I described above.
On the topic of what the average hn reader considers "satellite internet". Based on consumer experience with the absolute cheapest consumer grade ku and Ka band vsat terminals in the usa, they almost 99% of the time mean Hughesnet or viasat/wildblue. Very few people actually have money or reason to lay their hands on real two way satellite stuff. Generally if you work for a company that needs something like a $3500/month link for a mine in Nunavut you're paying some third party experts to implement it.
> 3.8 meter ku band dual linear polarity compact cassegrain dish > a 5.5m dish
Are these receiver sizes in the range of what Starlink expects from users? I had in mind something more like a DirectTV dish. If the consumer dish sizes are indeed smaller, how does that affect the discussion about jitter and other aspects of link quality?
The starlink cpe is indeed in the category of directv dish sized.
With only a tiny bit of the relevant subjects under my belt, I would expect the metrics to be significantly worse using a much smaller dish. But it seems like you're suggesting Starlink could achieve similar metrics on a smaller dish? I'm curious how that works. Apologies if I'm misunderstanding.
Note that may be true for one bird in Afghanistan, but the path length isn't exactly the same everywhere, nor are the delays of all modulaiton schemes.
For instance our site in Germany was more than 15ms away by fiber from DE-CIX stuff in Frankfurt. The site in Singapore had better latency to any cdn or hosting operation that was in Singapore itself, but much worse if traffic had to find its way through the terrestrial submarine fiber internet to other places. End to end latency between Afghanistan-germany-seattle or Afghanistan-singapore-seattle was considerably different.
Yes, the position of the two end point sites on the sphere that is the earth will have slightly different speed of light times to a satellite in geostationary orbit. But 99% of that latency in the above 490ms figure is the extreme distance of the satellite, and not whether or not one site is slightly further down the sphere, as viewed from a theoretical observer sitting on the satellite.
It is also worth noting that modern scpc/mcpc modems can be intentionally configured to use more computationally "expensive" FECs, in a trade off of slightly more bps/Hz squeezed into a given sized slice of transport kHz, at the expense of a few ms higher latency. This is a network engineering design decision made when a link is first brought online. Modern modems and ASICs, FPGAs have made this less difficult than it used to be. If I can use an analogy for software developers, this is sort of like the speed vs file size trade off choice made between using gzip, bzip2, xzip, at different settings and compression levels.
Path length varies from ~34500km to ~42000km (~115ms to ~140ms). So, assuming an ideally placed teleport, from 115 * 4 = 460ms to 115 * 2 + 140 * 2 = 510ms speed of light delay on round trips. This is why I said I found 492ms to be an amazingly over-precise number. Just call it 500ms. :P
Let me see if I can answer some of this.
We were on a 1:1, eventually. Tried the 8:1 (or something like that) at first and it was terrible.
Latency was almost always, like you said, almost exactly 495ms at the link. However, we'd suffer transient jitter that directly correlated to weather in the atmosphere. Essentially, satellite communication is a laser (parabolas ftw), so this made sense at the time. Correlation doesn't necessarily mean causation, but it was annoyingly correlated.
I only made this correlation because my "day job" in the military had a lot to do with radar, microwave antennas, and optics. So, I'd observe the refraction personally (being able to see mountains as clutter for example, when the radar rays were pointed well over the mountains) and then notice the observed jitter on the network. Not to mention people coming to tell me how unusable Skype was at the moment and could I fix it please?
No offense, but you probably don't see this at your level as a network engineer unless you're "boots on the ground." When you're there and easily accessible, people will come complain to you when any and every little thing that goes wrong and "their wife is threatening to leave them and they can't even have a conversation, omg, fix it right now plz."
In a more corporate setting, people are far less likely to complain about transient issues because they fix themselves before the complaint can be made, or if they do, it's easy to dismiss as other issues.
To be honest, we had those other issues as well. QoS was a thing, and constantly battling bittorrent clients and their users was also a thing. But those issues were usually only during the evenings when lots of people were trying to call home and get on Facebook. I'm talking about when none of those issues were at play, such as late at night or during the day.
The UDP solution was directly related to periods of saturation, specifically I built a custom transparent proxy forked from Squid. It's sole job was to act as a mediator between a server back in the US that did a few things:
1. On the Afghanistan side, it would wrap the HTTP(S) request packet and send it via a TCP control channel to the server in the US.
2. On the US side, it would do some image compression (if applicable) and other things to the response to make it smaller (in bytes).
3. On the US side, it would also keep track of user flows and how much bandwidth each user got, and spit back UDP packets, reserving a dynamic % of bandwidth for control and unregulated packets. There was also an open TCP control connection for sending back smaller things that fit within a few packets.
4. On the Afghanistan side, it would put all these packets together to reform the stream, keeping track of any dropped packets (rare) and then forwarding the completed stream (or once the buffer was full) to the user at 100mbps. If any were dropped, it sent back a bitmap over the open TCP connection to have those frames retransmitted via TCP and then inserted into the stream before going to the client.
It worked fantastic for things like youtube, since the subscriber could make full use of their subscribed bandwidth with a half-second delay after clicking a link. The only things that didn't use that system were things like Skype. We never really had transport-related congestion issues after this was built as a hack, over several weeks. To be honest, it probably would have worked just fine by forwarding over already open TCP connections to the back-end server and giving each subscriber a dedicated stream. However, time in the desert is a premium and once something works good-enough, it pretty much stays that way forever.
> If the person I'm replying to isn't willing or able to clarify whether they were in a contended, or dedicated network, or specify the gear, it's hard to say exactly why they're seeing jitter. But I can quite confidently say that excursions above 495ms are not something seen in a dedicated network.
It was over ten years ago at this point, so I don't remember the exact models of anything. I do remember it was ku-band, but that's not exactly helpful. :) I sold everything off after getting sent back to the US for a medical discharge. The source was also lost in that pretty crazy time of my life, mostly due to the laptop ceasing to exist. The code still probably exists in SVN somewhere (Google Code, probably).
Even when you can see a mountain multiple times at once, the apparent distance to the mountain is going to have almost no variance as it shimmers around.
Everybody can be wrong about something, no matter how smart / credentialed / experienced they are.
That said unless we're talking about lasers air refraction doesn't usually impact RF(there is tropospheric ducting but that's something different).
Latency as experienced by users might be much worse, however, if suddenly 300Mbps of traffic is trying to be shoved through a 40-75Mbps link, for a period of several minutes or more.
From the point of view of an end user on a LAN at a site dependent on two way satellite it may not be possible for them to determine exact why they're seeing packet loss.
Ultimately, the latter is the kind of latency the user cares about.
I like that to you only one is a random guy on the internet. And I should know, I'm the president of the World Liking Things club, and you random guy on the internet have no business liking things like I do.
Only on HN
But my knowledge of networks is fairly rudimentary and old now and now that I think about it, pings would get sudden spikes in the rain too and that's just a lower layer than I'm talking about. hmm. I was also a child then so I might be misremembering.
For an example of a modern radio google aviat wtm4200.
For something like the af5xhd, it's a great little thing for its price, and with good quality antennas at both ends can make a very effective bridge. But for more serious things? No. It's so bad they haven't even been able to implement snmpv2 on their $1000+ radios. The af11 is so bad it only uses a 56MHz channel mask in an 80MHz channel.
No matter what manufacturer, serious ptp radios can be distinguished by two things: the presence of terminals for a direct -48vdc wiring connection and sfp/sfp+ ports.
At the very low end of cost, ubnt's 802.11ac chipset ptp bridge radios are great and very cost effective, I'm talking about the powerbeam ac series, nanobeam and similar. For when you want to buy a few hundred bucks of stuff on your credit card to link a few buildings together quickly. For anyone looking at that category of stuff they should also look at mikrotik and mimosa radios.
Edit: And, any really good blog series on this? so I know what to expect in such a course
Until very recently you have not been able to move more than 2400 bps through an individual iridium link, limiting it to very low data rate IP networking and single user at a time per device.
The military itself uses a lot more geostationary than you might think at first glance. They don't have their own iridium sized low earth orbit narrow band networks. There's many series of DoD satellites in geostationary, and they also lease a lot of commercial transponder capacity.
The other uplink was a solar powered 4G connection and despite several deep cycle batteries and a few 200 watt panels, the station did not get enough light to stay powered over those weeks.
We finally found a hilltop wireless ISP that can serve us. We’ll try to get Starlink too but I don’t expect that to work well during fire season. It’s tough to be offline for weeks at a time.
When dealing directly with UDP, one can implement different application-level logic for dealing with packet loss: "NAK" packets, different backoff strategies, and even forward error correction.
TCP is designed for encapsulation in IP, this is why it is called TCP/IP. And UDP is very similar to IP, only difference is that UDP gives you source and destination port numbers and a payload checksum, where IP only gives you a protocol numbers and non-checksummed data. None of these "features" is going to have any effect on how TCP is delivered.
So the only effect that use of TCP over UDP would have here is:
- increase the overhead used for headers, thereby decreasing the MSS (segment size)
- "cloak" the TCP traffic as UDP and thereby prevent any routers and/or middleboxes [2] from applying special treatment to TCP (i.e. traffic shaping, buffering etc.)
Maybe the latter effect is what made a difference for the link in question.
[1] https://tools.ietf.org/id/draft-baset-tsvwg-tcp-over-udp-01....
[2] https://en.wikipedia.org/wiki/Middlebox
edit: formatting
But he wrote that he'd used "TCP over UDP". The first lines of the TCP over UDP spec already state
"It provides exactly the same congestion control, flow control, reliability, and extension mechanisms as offered by TCP." [1]
[1] https://tools.ietf.org/id/draft-baset-tsvwg-tcp-over-udp-01....
Source: I ran a small satellite ISP on a base in Afghanistan.