After writing a test script and letting it sit in a loop the bug became reproducible: 1152 times got a replication.
Instead of real observation, there's a lot of leaning on type theory, OOP, immutability, and other ideas that scratch some sort of neatness itch and/or advance the careers of opinionated language creators, but have little/no empirical support to prove they actually produce more reliable code.
Meanwhile formal techniques and languages that do have empirical support are sidelined and ignored.
Bugs like yours are not uncommon. If anything, because of Darwinian debugging - the obvious ones die first - you'd expect a good proportion of hard-to-reproduce bugs in any code base.
So they shouldn't be considered weird aberrations. I'm not sure why CS theory doesn't have a better handle on them.
Every anti-bug measure tends to work like yet another brush aimed at a wall. But most brushes overlap partially or even wholly with areas of the wall already painted with previous brushes. The only strategy that would really work would cover all of the wall in a very systematic fashion.
I once wrote an industrial control program, and used a time function that rolled over at midnight. To make matters worse, we never bothered to set the clocks on the computers, so it seemed like a completely random glitch, and was a devil to debug.
If so crash the program with a kill -9 set up to write the core to disk, then run gdb on the executable and reload that core file. Then run 'where'. This will tell you where you're hanging and how you got there as well as all the parameters to functions on the way down.
A ps -aux beforehand can give you another hint, look at the status letter.
Good luck!
The database person spent a month trying to figure out why there were entries with odd dates. Like one or two a month out of a few hundred thousand. Eventually figured out it wasn't his fault. His end was getting an occasional packet with a good crc and odd data. Most of the time the parser would kick those out. But a few were well formed enough to go through.
I chased it down eventually to a documentation error in the datasheet for the radio transceiver we were using. The datasheet very clearly stated that when a packet came in with a bad crc... nothing would happen. No end of packet interrupt. So I had to use a timer to detect a bad crc. (Okay whatever, I mean I hate those guys, but I've seen worse)
Turns out when the radio received a packet with a wrong/bad crc the radio silently reset the receiver and started waiting for a new packet. If a the radio received a packet before the timeout hammer came down then the driver would think the previously received packet was 'good'
Opps.
Was interesting because the symptom appeared at the farthest end of the system the tool graphing data for the customer and was caused by something at the other end. The error could have been literally anywhere in our system.
* gcc dies with internal compiler error when you try to compile with debug symbols.
By this time I've killed gcc, javac, ghc, and the .Net compiler. Worse, the ghc bug was the only one I could track.
* Security : customers acccounts leaked, identities stolen, bank account depleted
* Undetected data corruption : customers' data is corrupted and it has seeped into the backup stream, even the off-site one, and that is corrupted as well now.
* Rarely occurring simingly random concurrency related bugs that manifest themselves once a month or longer.
* Time : making assumptions about it, dealing with it distributed systems etc. Nobody thinks about it until it stops working or systems' time gets snapped back somehow.