Previous HN thread: https://news.ycombinator.com/item?id=8171956
We should be monitoring the rate of test failures -- per test -- in a timeseries and looking for periodicities. Just graphing them and seeing regular spikes will help, but having real numerics looking for approximate periodicities will catch this sort of thing (after a few months!).
But, as a general rule, I would try to brute force or fuzz your "inputs", a.k.a data out of your control. Random numbers, dates, user input, network packets, etc. Users and environments often act in unexpected ways and you want to make sure your bug finding isn't reduced to accidentally be working on the right day! :)
But the bigger issue was that since it would "fix itself" we never prioritized it and just left it alone.
But looking back, the system (automated tests) was functioning as designed. :) We just changed our "DateTime.now" to concrete dates.
When we decided to look into it when we had some downtime, we found out that it was a date bug that only reared its head on the 31st day of the month, hence only happening every two months.
I'm not sure bisect would've helped but still was really funny when we did find the issue.