Re:Going Big with TCP Packets
by toast0 on Thursday February 17, 2022 @11:02AM
(#30374552)
Your problem is highly likely to be in the first hop, or first few hops. Adjusting the MTU for the whole interface is the right thing to do there.
Historically, ICMP blackholes tended to be on long distance international transit, though. So maybe your ISP was 1500 MTU throughout and most of its transit and peering was too (or sent ICMPs anyway), but routes to Mordor were few and far between, so your ISPs transit provider used one that was IPIP tunneled (so effective MTU was 1480) and either didn't send ICMPs or was rate limited on sending ICMPs so it was as if they didn't send them. Everything would work OK to most of the internet, but if you made a connection to Mordor (or someone from Mordor made a connection to you), small packets make it through and large packets get dropped, and you get that kind of works but doesn't really work behavior.
In that situation, if you're in Mordor and most of the things you connect to aren't, it makes sense to set a smaller MTU. If you're not in Mordor, but sometimes connect to it, you want probing to happen for those connections, but setting a smaller MTU probably isn't the right thing (but, popular sites do it). If you're in Mordor and mostly connect to Mordor services, but sometimes the outside world, you again want probing sometimes.
But keep in mind, the probing really only works if both sides are doing it. It's hard to induce your TCP peer to send smaller packets if you sent a too big MSS.
All that said, my experience is that in recent times, it seems like problems are usually some ISP PPPoE CPE that isn't clamping MSS. International transit has gotten pretty good over the past 30 years.
Historically, ICMP blackholes tended to be on long distance international transit, though. So maybe your ISP was 1500 MTU throughout and most of its transit and peering was too (or sent ICMPs anyway), but routes to Mordor were few and far between, so your ISPs transit provider used one that was IPIP tunneled (so effective MTU was 1480) and either didn't send ICMPs or was rate limited on sending ICMPs so it was as if they didn't send them. Everything would work OK to most of the internet, but if you made a connection to Mordor (or someone from Mordor made a connection to you), small packets make it through and large packets get dropped, and you get that kind of works but doesn't really work behavior.
In that situation, if you're in Mordor and most of the things you connect to aren't, it makes sense to set a smaller MTU. If you're not in Mordor, but sometimes connect to it, you want probing to happen for those connections, but setting a smaller MTU probably isn't the right thing (but, popular sites do it). If you're in Mordor and mostly connect to Mordor services, but sometimes the outside world, you again want probing sometimes.
But keep in mind, the probing really only works if both sides are doing it. It's hard to induce your TCP peer to send smaller packets if you sent a too big MSS.
All that said, my experience is that in recent times, it seems like problems are usually some ISP PPPoE CPE that isn't clamping MSS. International transit has gotten pretty good over the past 30 years.