News for hackers. Stuff that matters.
Re:I discovered caching CDNs were throttling my everyday bro... by londons_explore on Thursday November 23, 2023 @07:50AM (#38392213)
I think this is just what you see with typical ISP "traffic shapers".

They try to limit bandwidth to video sites, but since most video traffic is transferred by HTTPS these days they end up just making a massive list of IP's which look like they might be sending video data and dropping some percentage of traffic to those IP's. Most CDN's are probably on the list.

End result is most video sites drop back to SD rather than HD.

If you do a speedtest, it will come out as fast. If you VPN, that will also be fast.

The IP range has nothing todo with it - it is the route the packets traverse and what the packets look like when they pass the shaper device that matters.

You could theoretically find out which device on the path is doing the dropping by manipulating the TTL of packets in a live TCP session and seeing when you get back TTL exceeded messages.

Re:I discovered caching CDNs were throttling my everyday bro... by supriyo-biswas on Thursday November 23, 2023 @08:13AM (#38392387)
There’s fast.com and speed.cloudflare.com to get realistic speed measurements since there’s no way* the ISP can tell traffic to these sites vs. video traffic to Netflix/Cloudflare.

*Technically some ISPs can (and probably do) look at the ServerNameIndication to determine hostname and therefore whether they’re video flows or not, but I’ve not heard any real world examples of this being done.

Re:I discovered caching CDNs were throttling my everyday bro... by paulryanrogers on Thursday November 23, 2023 @08:30AM (#38392525)
ESNI to the rescue?
Re:I discovered caching CDNs were throttling my everyday bro... by cesarb on Thursday November 23, 2023 @08:36AM (#38392579)
> ESNI to the rescue?

IIRC, it's now called ECH, because AFAIK the focus changed from "encrypt the SNI" to "encrypt the whole Client Hello".

Re:I discovered caching CDNs were throttling my everyday bro... by crotchfire on Thursday November 23, 2023 @09:26AM (#38393091)
Uh, no, the focus changed to don't encrypt the outer SNI but pretend we're encrypting something that will prevent filtering, even though it won't.
Re:I discovered caching CDNs were throttling my everyday bro... by lxgr on Thursday November 23, 2023 @09:49AM (#38393308)
Isn't the outer SNI something that can be deduced from the IP address anyway?
Re:I discovered caching CDNs were throttling my everyday bro... by infotogivenm on Thursday November 23, 2023 @09:48AM (#38393300)
Can you elaborate? I have not kept up with the eSNI/ECH stuff... How will filtering still be possible?
Re:I discovered caching CDNs were throttling my everyday bro... by jacooper on Thursday November 23, 2023 @09:52AM (#38393338)
Do you have more details on that ?
Re:I discovered caching CDNs were throttling my everyday bro... by avipars on Thursday November 23, 2023 @10:43AM (#38393968)
Computer Scientists pick the best names for things
Re:I discovered caching CDNs were throttling my everyday bro... by bsoft16385 on Thursday November 23, 2023 @08:39AM (#38392606)
Modern deep packet identification gear looks at packet size and pacing to distinguish video traffic patterns. It is possible to fool these systems, but it is not trivial.
Re:I discovered caching CDNs were throttling my everyday bro... by Waterluvian on Thursday November 23, 2023 @08:46AM (#38392679)
Any reason fast.com couldn’t stream a video as the test content?
Re:I discovered caching CDNs were throttling my everyday bro... by nippoo on Thursday November 23, 2023 @10:08AM (#38393521)
Yes, this is exactly what it does, and it's a very clever trick by Netflix to discourage their traffic from being throttled!
Re:I discovered caching CDNs were throttling my everyday bro... by pests on Thursday November 23, 2023 @08:49AM (#38392717)
I think it does pull a real video, I'm not sure its pulled in the same way the actual Netflix video player would do it.
Re:I discovered caching CDNs were throttling my everyday bro... by tuetuopay on Thursday November 23, 2023 @09:44AM (#38393250)
you wouldn't pull at gigabits/s bandwidth when casually watching netflix
Re:I discovered caching CDNs were throttling my everyday bro... by londons_explore on Thursday November 23, 2023 @10:04AM (#38393480)
dunno about netflix, but youtube usually does. They send you a chunk of data as fast as possible, and then send nothing till your client requests the next chunk.

By doing that, they can get a good estimate of your connections available bandwidth, which is needed for the decision whether to automatically switch to a higher/lower quality feed.

It also means they can use 'dumb' servers which don't do any application-specific logic for throttling.

Re:I discovered caching CDNs were throttling my everyday bro... by chatmasta on Thursday November 23, 2023 @10:58AM (#38394162)
Recently I've noticed my phone disconnecting from my Wi-Fi router (a relatively underpowered device running OpenWRT and WireGuard) when starting YouTube videos. I guess this could explain it, if the router is hiccuping on that initial blast of data. I also use a Safari extension that lets me choose the YouTube video quality, so that might affect it if I'm not giving the feedback they expect for "auto" quality selection.

(The issue is not the router, though - this doesn't happen on my laptop. I think my phone is just more aggressive in assuming the Wi-Fi is down and it's better to switch to cellular. But couple that with my phone's policy to use VPN on cellular, and the switch becomes much higher friction. I tried simply disabling cellular data, but then it's even worse because every time the phone disconnects from Wi-Fi it pops an alert telling me to enable cellular data.)

Re:I discovered caching CDNs were throttling my everyday bro... by tuetuopay on Thursday November 23, 2023 @10:19AM (#38393651)
makes sense, but in that case it's quite easy to distinguish between a burst of traffic of <1s to a speedtest that last tens of seconds
Re:I discovered caching CDNs were throttling my everyday bro... by crotchfire on Thursday November 23, 2023 @09:25AM (#38393080)
Are you kidding?

Snooping like this is EXACTLY why we still have unencrypted ServerNameIndication, even to this day in ECH (which still leaves the outer SNI in the clear).

Re:I discovered caching CDNs were throttling my everyday bro... by lxgr on Thursday November 23, 2023 @09:52AM (#38393341)
What does the outer SNI tell a hypothetical filtering/traffic shaping ISP that the IP address doesn't already?
Re:I discovered caching CDNs were throttling my everyday bro... by londons_explore on Thursday November 23, 2023 @10:24AM (#38393712)
lots of sites often just are connections to cloudflare/akami/fastly these days. Ie. one IP, millions of websites. The ISP/feds need SNI to tell what you're up to.
Re:I discovered caching CDNs were throttling my everyday bro... by lxgr on Thursday November 23, 2023 @10:54AM (#38394111)
Yes, but the outer SNI (with ECH) in that case would just say "Cloudflare", "Akamai" etc., if I understand it correctly.
Re:I discovered caching CDNs were throttling my everyday bro... by gruez on Thursday November 23, 2023 @09:05AM (#38392865)
>but since most video traffic is transferred by HTTPS these days they end up just making a massive list of IP's which look like they might be sending video data and dropping some percentage of traffic to those IP's.

You really don't when every modern browser sends SNI, which exposes the full domain name it's trying to connect to.

Re:I discovered caching CDNs were throttling my everyday bro... by karmakaze on Thursday November 23, 2023 @10:15AM (#38393605)
In most cases an ISP is also doing DNS resolution so would already be able to pretty accurately reverse map IPs to hostnames without reverse records.
Re:I discovered caching CDNs were throttling my everyday bro... by lxgr on Thursday November 23, 2023 @01:35PM (#38395927)
That's much easier to fix (by using DoH or DoT) than getting ECH widely deployed, though. Importantly, it can be done unilaterally by each user and doesn't need cooperation on both sides of the connection.

Many browsers these days support using DoH without changing any OS settings.

Re:I discovered caching CDNs were throttling my everyday bro... by karmakaze on Thursday November 23, 2023 @06:59PM (#38399206)
That's not actually a fix. As long as some other subscriber does the DNS lookup, then the ip address will be associated with the name that was looked up. If you happen to use a different resolver and get the same address, there's no difference.
Re:I discovered caching CDNs were throttling my everyday bro... by lxgr on Thursday November 23, 2023 @08:05PM (#38399682)
If the IP address belongs to a cloud load balancer or CDN, that’s still much better than hostname-level detail. Even for subdomains it helps (think somebody.bloghoster.tld).

And ECH will still reveal the outer SNI, apparently.

Re:I discovered caching CDNs were throttling my everyday bro... by guappa on Thursday November 23, 2023 @09:29AM (#38393132)
I got called a paranoid a couple of years ago, because I said that my provider slows down long downloads, since every download starts fast, but after a few megabytes they become slower and slower.
Re:I discovered caching CDNs were throttling my everyday bro... by kaetemi on Thursday November 23, 2023 @01:01PM (#38395525)
If you set your router to limit bandwidth below that initial burst speed, to your actual connection speed, you'll get consistent fast speed.

The burst mechanism of your provider possibly squeezes below your normal speed to compensate for the burst, but ends up confusing TCP flow.

Those burst and squeeze cycles can cause some undesirable bandwidth oscillations.

Re:I discovered caching CDNs were throttling my everyday bro... by lxgr on Thursday November 23, 2023 @01:32PM (#38395899)
Any TCP congestion control algorithm worth its salt should be able to adapt to varying path capacity these days, especially given how common traffic shaping with bursting is in residential connections.
Re:I discovered caching CDNs were throttling my everyday bro... by kaetemi on Friday November 24, 2023 @07:10PM (#38409807)
You can safely assume the bursting itself is badly implemented on shitty ISP modems and causes spikes of high packet loss and latency when it switches.

Say you're bursting 1gbps on a connection with 100ms latency. That connection squeezes to 20mbps suddenly. In the best case in those 100ms that the sender cannot even react to the change, we receive 10mbytes. Those 10mbytes that now have to be throttled away either get dropped or end up in a buffer. Now you either have a 5 second backlog latency or massive packet loss for 100ms while this resolves. And things overcompensate. And worst case, you end up in a oscillation pattern, the throttle goes off, and you burst again.

This happens and is measurable.

Re:I discovered caching CDNs were throttling my everyday bro... by callalex on Thursday November 23, 2023 @01:00PM (#38395518)
That’s not paranoia, it’s bog standard traffic shaping that ISPs have been doing since the 90’s or even earlier.
Re:I discovered caching CDNs were throttling my everyday bro... by guappa on Friday November 24, 2023 @01:47AM (#38401232)
Eh, others in the discussion claimed it would be completely impossible for an ISP to do this thing, due to how network protocols work.

Anyway, annoying… at least http supports resuming, so it is possible to kill and restart manually, or use aria2c to circumvent it.

Re:I discovered caching CDNs were throttling my everyday bro... by kaetemi on Friday November 24, 2023 @07:18PM (#38409860)
ISP controlled modem counts your incoming data in a short rolling window. If it exceeds a value, it throttles below your normal speed and buffers or drops incoming packets to slow things down, until the bandwidth budget is compensated for (and the dropped packets are all counted, so it could be stuck there longer than anticipated), then switches to normal speed until traffic goes below a threshold, then switches back to full speed burst. Worst case the flow control reacts slower than the modem burst switching and you end up in an oscillating pattern.

The trick is just to do your own rate limiting without burst at the right limit, that bypasses whatever shitty burst implementation your ISP modem has and weird shit won't happen anymore.

Re:I discovered caching CDNs were throttling my everyday bro... by lxgr on Thursday November 23, 2023 @09:59AM (#38393420)
Are you using cable by any chance?

This is a pretty standard feature of DOCSIS, I believe. It allows bursting above the provisioned connection speed for a few seconds when there's capacity on the shared network segment.

Re:I discovered caching CDNs were throttling my everyday bro... by bobdvb on Friday November 24, 2023 @05:09AM (#38402308)
It doesn't have to be traffic shaping, it can be something as simple as limited peering capacity. Hanlon's Razor applies, this isn't necessarily malicious capping.

Most ISPs don't have one fat pipe to this magical "internet" place, they have multiple pipes going different places and not all of them are even listed in peeringdb, so their route to a particular CDN might be over private peering which is limited in size (e.g. 1-10Gb). Sure, then excess traffic might go over public peering but that might also then get saturated, or the ISP might force traffic over the private peering to keep the public peering clear for general traffic.

If you use a VPN then the traffic will come from a different route and different peering, thus be subject to the capacity of that route. What the OP probably wants is for Zen to have better peering with Akamai (they already have some announced routes).