News for hackers. Stuff that matters.
Re:TCP Connection Timeout Mystery by Existing4190 on Monday March 25, 2024 @02:39PM (#39819813)
I'm suspicious about the IP 169.150.221.147 My guess: there is some misconfigured bogons IP filter and instead of 169.254.0.0/16 (rfc3927) there is something like 169.0.0.0/8 configured to be blocked on some firewall

I once was a customer of an ISP that mistakenly blocked the whole 192.0.0.0/8 net, which caused some confusion, but they fixed it after I pointed it out.

Re:TCP Connection Timeout Mystery by js2 on Monday March 25, 2024 @02:58PM (#39820035)
But then why would the ICMP echo/reply (ping) be allowed through? And how is the initial syn/ack and reply getting through? It's only the second ack that's getting (apparently) blocked.
Re:TCP Connection Timeout Mystery by Existing4190 on Monday March 25, 2024 @03:14PM (#39820218)
You are right. My comment can't solve the whole story.

Still, some middlebox/stateful firewall/etc. messing with 169.0.0.0/8 is plausible.

Re:TCP Connection Timeout Mystery by xnyan on Monday March 25, 2024 @04:54PM (#39821159)
Actually I think you might still be right. Ping uses ICMP, which is almost never blocked in my experience. I learned that because early in my career I too assumed a successful ping = TCP and UDP also work.
[deleted]
Re:TCP Connection Timeout Mystery by rstuart4133 on Monday March 25, 2024 @07:01PM (#39822227)
Similar happened to me not so long ago. One day a junior admin asked me to diagnose who ssh to a box started hanging. After a bit of diagnosis it became evident TCP's 3 way handshake got through, and then it all stopped. No normal network behaves like this.

My answer was "some network admin is having fun with a middleware box", you will have to speak with them. They did, and the response that came back is "we are moving to a network with real security". Access was restored. It was a Palo Alto NGFW as others have mentioned.

IMHO the industry looking for way to move high priced gear, and they convince someone with a sparrow problem with barely knows how to handle an air rifle to buy bazooka's. A bit of collateral damage is to be expected....

Re:TCP Connection Timeout Mystery by js2 on Monday March 25, 2024 @06:44PM (#39822078)
Filtering + established connection tracking:

https://news.ycombinator.com/item?id=39822214

Re:TCP Connection Timeout Mystery by xyst on Monday March 25, 2024 @03:13PM (#39820201)
Probably because the firewall rule only includes TCP/UDP. ICMP is often not blocked, in my experience.
Re:TCP Connection Timeout Mystery by js2 on Monday March 25, 2024 @06:07PM (#39821780)
That doesn't explain the initial SYN/ACK and reply ACK which are part of the TCP establishment and that we see getting through.
Re:TCP Connection Timeout Mystery by dunham on Monday March 25, 2024 @06:31PM (#39821979)
It's been a very long time since I've diagnosed something like this, but I've had problems in the past when the MTU is smaller than the default and ICMP is blocked (interfering with path MTU discovery). Often IPSec or some other tunneling was involved. The initial packets got through but as soon as a full packet was sent it was dropped.

EDIT - I've now scrolled down in HN and saw that this was ruled out.

Re:TCP Connection Timeout Mystery by xnyan on Monday March 25, 2024 @04:49PM (#39821114)
ICMP (the protocol ping uses) is a totally separate protocol from TCP and UDP. Blocking ICMP can break of lot of things and offers no real benefits outside of a handful of specific edge cases.

BTW your assumption "a successful ICMP ping = TCP and UDP work" is an extremely common one that I too had before I was taught otherwise.

Re:TCP Connection Timeout Mystery by js2 on Monday March 25, 2024 @06:04PM (#39821749)
> BTW your assumption

I did not assume. The comment to which I was responding suggested it was the destination IP that was the problem. Generally (but not always) an IP filter would be applied irrespective of protocol. I also pointed out that the initial SYN and reply SYN/ACK are getting through the hypothesized bogon filter and those are part of TCP. I don't think the bogon filter is a hypothesis that fits the evidence.

ETA: but adding connection state tracking + a filter does make sense https://news.ycombinator.com/item?id=39822214

Re:TCP Connection Timeout Mystery by xnyan on Wednesday March 27, 2024 @01:02PM (#39841696)
> Generally (but not always) an IP filter would be applied irrespective of protocol

I've never seen this (all protocols by default) in any environment I've personally worked with, but perhaps I've had a unique experience.

Re:TCP Connection Timeout Mystery by whirlwin on Monday March 25, 2024 @05:09PM (#39821298)
> Blocking ICMP can break of lot of things and offers no real benefits outside of a handful of specific edge cases.

Are you referring to local networks only?

It's very common to not allow ICMP by defaul to workloads in the cloud, e.g. in AWS.

Re:TCP Connection Timeout Mystery by fragmede on Monday March 25, 2024 @05:26PM (#39821435)
That's likely to be an implementation detail of how they've implemented TCP routing across a large fabric.
Re:TCP Connection Timeout Mystery by Hikikomori on Monday March 25, 2024 @06:31PM (#39821977)
AWS doesn't decide or even care about this, customers configure security group rules for their own services. Nothing is allowed by default, so if you want ICMP you would need to allow it, most font bother because it's not that helpful in a cloud environment (can just monitor the TCP port instead and get similar information).
Re:TCP Connection Timeout Mystery by immibis on Monday March 25, 2024 @06:43PM (#39822073)
This explains why some people have problems with IPv6 - if you block IPv6 Control Messages, then it will only work sometimes.
Re:TCP Connection Timeout Mystery by pajko on Wednesday March 27, 2024 @01:03PM (#39841700)
Fragmented packets won't work without ICMP.

Edit: here's a good page about the effects of disabling ICMP: https://www.rimscout.com/why-you-should-not-block-icmp/

Also there's some blackhole detection or how is it called.

However it's OK to block _parts_ of the ICMP protocol for security reasons, like echo and reply.

Re:TCP Connection Timeout Mystery by Thev00d00 on Monday March 25, 2024 @02:42PM (#39819858)
Yeah, this was my immediate thought, someone has made the 169.254 block too large somewhere.
Re:TCP Connection Timeout Mystery by axus on Monday March 25, 2024 @03:19PM (#39820279)
They should go to http://169.150.211.2 and see if that gets blocked. I get a "Welcome to nginx!" page there.
Re:TCP Connection Timeout Mystery by WirelessGigabit on Friday April 05, 2024 @04:54PM (#39947090)
That's weird. I get nothing there.
Re:TCP Connection Timeout Mystery by dunham on Monday March 25, 2024 @06:25PM (#39821928)
My employer did something like that once, and it took out access to github.