You are right. My comment can't solve the whole story.

Still, some middlebox/stateful firewall/etc. messing with 169.0.0.0/8 is plausible.

Actually I think you might still be right. Ping uses ICMP, which is almost never blocked in my experience. I learned that because early in my career I too assumed a successful ping = TCP and UDP also work.

Similar happened to me not so long ago. One day a junior admin asked me to diagnose who ssh to a box started hanging. After a bit of diagnosis it became evident TCP's 3 way handshake got through, and then it all stopped. No normal network behaves like this.

My answer was "some network admin is having fun with a middleware box", you will have to speak with them. They did, and the response that came back is "we are moving to a network with real security". Access was restored. It was a Palo Alto NGFW as others have mentioned.

IMHO the industry looking for way to move high priced gear, and they convince someone with a sparrow problem with barely knows how to handle an air rifle to buy bazooka's. A bit of collateral damage is to be expected....

Filtering + established connection tracking:

https://news.ycombinator.com/item?id=39822214

Probably because the firewall rule only includes TCP/UDP. ICMP is often not blocked, in my experience.

That doesn't explain the initial SYN/ACK and reply ACK which are part of the TCP establishment and that we see getting through.

It's been a very long time since I've diagnosed something like this, but I've had problems in the past when the MTU is smaller than the default and ICMP is blocked (interfering with path MTU discovery). Often IPSec or some other tunneling was involved. The initial packets got through but as soon as a full packet was sent it was dropped.

EDIT - I've now scrolled down in HN and saw that this was ruled out.

ICMP (the protocol ping uses) is a totally separate protocol from TCP and UDP. Blocking ICMP can break of lot of things and offers no real benefits outside of a handful of specific edge cases.

BTW your assumption "a successful ICMP ping = TCP and UDP work" is an extremely common one that I too had before I was taught otherwise.

> BTW your assumption

I did not assume. The comment to which I was responding suggested it was the destination IP that was the problem. Generally (but not always) an IP filter would be applied irrespective of protocol. I also pointed out that the initial SYN and reply SYN/ACK are getting through the hypothesized bogon filter and those are part of TCP. I don't think the bogon filter is a hypothesis that fits the evidence.

ETA: but adding connection state tracking + a filter does make sense https://news.ycombinator.com/item?id=39822214

> Generally (but not always) an IP filter would be applied irrespective of protocol

I've never seen this (all protocols by default) in any environment I've personally worked with, but perhaps I've had a unique experience.

> Blocking ICMP can break of lot of things and offers no real benefits outside of a handful of specific edge cases.

Are you referring to local networks only?

It's very common to not allow ICMP by defaul to workloads in the cloud, e.g. in AWS.

That's likely to be an implementation detail of how they've implemented TCP routing across a large fabric.

AWS doesn't decide or even care about this, customers configure security group rules for their own services. Nothing is allowed by default, so if you want ICMP you would need to allow it, most font bother because it's not that helpful in a cloud environment (can just monitor the TCP port instead and get similar information).

This explains why some people have problems with IPv6 - if you block IPv6 Control Messages, then it will only work sometimes.

Fragmented packets won't work without ICMP.

Edit: here's a good page about the effects of disabling ICMP: https://www.rimscout.com/why-you-should-not-block-icmp/

Also there's some blackhole detection or how is it called.

However it's OK to block _parts_ of the ICMP protocol for security reasons, like echo and reply.