Re:AT&T Fiber in the SF Bay Area is flipping bits
by watt on Tuesday December 08, 2020 @04:28AM
(#25343422)
Wasted an opportunity to implement code that would detect and handle poison-pill messages. Those will happen in any system where queue is involved and there always needs to be an escape hatch to get rid of them. Deleting the queue is too extreme.
Re:AT&T Fiber in the SF Bay Area is flipping bits
by miahi on Tuesday December 08, 2020 @12:38PM
(#25348607)
Deleting the queues is an operational decision that I made to be able to put the system back online after the network device was replaced (the important part was the uptime/SLA). From a quick analysis of the logs the percentage of bad messages was ~90% (there was a ~50% chance that the original "touched" bit was 0 so no change was done, but the messages had multiple "touched" bits at fixed intervals).
There was an escape hatch, but the conditions to hit it were a bit complex. Implementing new message filtering of this kind at 2AM while the system was down was not feasible.
On the application side the effects were quite bad, as the data was mainly XML and, depending on where the bit was flipped, it could impact the data or the XML structure. The data had its own CRC/hash, so the packets were cleanly rejected by the application. Unfortunately the XML library from the message queue engine and the ESB we were using did not like at all when the bit flipping occurred in the XML tags (it seems fuzzing tests were not done at that point) so the message processing got stuck and we kept getting bad messages in the queues. Even worse, the queues could not be cleaned with the normal procedures because the application wanted to first display info about the messages inside - and that failed.
The network debug was non-trivial because of that header consistency - the network devices did not report any kind of packet issues, so we had to sniff the different network segments to identify the culprit. From the application point of view, we had to delete the whole message queue storage to get rid of the bad messages, and let the application handle the rest (luckily it was designed with eventual consistency and self-healing).