News for hackers. Stuff that matters.
Re:Yesterday, my program worked. Today, it does not. Why? (1... by atonse on Wednesday July 19, 2017 @09:03AM (#14803954)
We once had a bug that would cause a failure in our test suite, but very rarely. It was so rare that it would just "go away" and not show up for weeks. So we thought it was odd (maybe bad test data, etc) but moved on.

When we decided to look into it when we had some downtime, we found out that it was a date bug that only reared its head on the 31st day of the month, hence only happening every two months.

I'm not sure bisect would've helped but still was really funny when we did find the issue.

Re:Yesterday, my program worked. Today, it does not. Why? (1... by brainfire on Wednesday July 19, 2017 @01:46PM (#14806260)
Re:Yesterday, my program worked. Today, it does not. Why? (1... by adrianratnapala on Wednesday July 19, 2017 @09:52AM (#14804253)
Thanks for the anecdote, because it gives me a new idea.

We should be monitoring the rate of test failures -- per test -- in a timeseries and looking for periodicities. Just graphing them and seeing regular spikes will help, but having real numerics looking for approximate periodicities will catch this sort of thing (after a few months!).

Re:Yesterday, my program worked. Today, it does not. Why? (1... by bArray on Wednesday July 19, 2017 @10:11AM (#14804366)
Unless you are writing date specific code or have seen this bug arise, I probably wouldn't bother. Time bugs will likely creep in regardless, for example how would you find a bug that happens one month a year? How would you find a bug once every four years? Sometimes human knowledge/experience can't be tested out of a system. All test failures probably warrant investigation of some type.

But, as a general rule, I would try to brute force or fuzz your "inputs", a.k.a data out of your control. Random numbers, dates, user input, network packets, etc. Users and environments often act in unexpected ways and you want to make sure your bug finding isn't reduced to accidentally be working on the right day! :)

Re:Yesterday, my program worked. Today, it does not. Why? (1... by michaelcampbell on Wednesday July 19, 2017 @09:13AM (#14804009)
Was this the kind of test that only ran once a day?
Re:Yesterday, my program worked. Today, it does not. Why? (1... by atonse on Wednesday July 19, 2017 @09:58AM (#14804283)
It wasn't, but it was the sort of thing where we never modified that test or even a related area, so it just wouldn't make sense that the test would fail.

But the bigger issue was that since it would "fix itself" we never prioritized it and just left it alone.

But looking back, the system (automated tests) was functioning as designed. :) We just changed our "DateTime.now" to concrete dates.

Re:Yesterday, my program worked. Today, it does not. Why? (1... by scorcher on Wednesday July 19, 2017 @11:47AM (#14805146)
Thats where property based tests come in very useful