Getting NSOC on-board is a big deal. After all, they deal with the entire VA network with 100,000+ employees. If you think about it from their perspective, why is USDS' TCP connections so special?
I just wanted to politely point out though, in this case, I think there should have been an indications of a network failure in this analysis early on, from the standpoint that TCP frames were sent to the server which were not acknowledged. This would depend on the point where you capture the traffic naturally, but the lack of acknowledgement would be a strong indicator that traffic is not reaching the server, or that replies are not reaching your capture point.
So while the TS Val may be the cause of the drops, I think the packet drops should have stood out when seeing the traffic being black holed, and likely the same segments getting re-transmitted continuously.
And for anyone out their who thinks this is easy to catch, I'd say this is very easy to miss, because you need to have a good understanding of how TCP works in the first place, to know what not working looks like.
- Inserting "sleep 300" into the startup process.
- Adding a cronjob to reboot the servers once a week.
I kid, but I'm sure we've all seen hacks like this.
1) Connections hanging over a frame relay network that one day started dropping packets over a certain size. Work-around was adjusting the MTU until I was able to convince the frame relay network operator that something was broken in their network. Initially it was confusing because an interactive telnet session over the network would work fine till you did something like "ls -l" or tried to read a man page which generated enough text to send a full size packet, then the connection would hang.
2) Unable to reach a Verizon e-mail paging gateway but only when connecting from a Linux box. An OS X box on the same network as the Linux box could reach the gateway fine. Turned out Verizon had a firewall rejecting connections where the ECN bit was set. Linux was setting ECN, OS X was not.
3) Solaris box A could initiate a connection to box B, but not the other way. After A talked to B, B could then talk to A, but only for a short period. Someone had deleted A's own MAC from A's ARP table, so A wasn't replying to ARP requests for itself. But if A connected to B, B would keep A's MAC in its own table till it timed out after which B couldn't initiate connection to A any more.
4) All manner of misconfigurations over the years where you learn to recognize the symptoms: misconfigured netmask size; misconfigured duplex; duplicate IP address on same network. You rarely see these any more.
5) The infamous 500-mile e-mail. :-)
6) And my favorite - https://www.pagerduty.com/blog/the-discovery-of-apache-zooke...