News for hackers. Stuff that matters.
Re:Spark joy by running fewer tests by viraptor on Thursday June 11, 2020 @08:42PM (#23494795)
How does your zero tolerance policy work when you can't find a reason the rest fails? I.e. you have critical functionality you test, but fail one test run in 1000. You can spend 2 days on the issue and not find a solution.

Sinking more time into it doesn't really help anyone much and won't offset any time saving in a realistic future. On the other hand, if you actually remove the test, you risk messing up a critical transaction.

Re:Spark joy by running fewer tests by maccard on Friday June 12, 2020 @04:45AM (#23497306)
If you have a critical function failing testing, you need to figure out the root cause, whether it takes 2 days or 2 months. 2 days is nothing - I've spent more than that on UI polish bugs, to not be able to dedicate 5x that time to fixing critical functionality, something is desperately wrong.
Re:Spark joy by running fewer tests by majkinetor on Saturday June 13, 2020 @08:05AM (#23508827)
Viraptor is right. Your "solution" is far from unversally acceptable. If you came to me as developer telling me that you did 2 months debug on test that fails randmoly once in a week and you didn't do much else, I would probably show you the dor as a team lead. We need to put things into perspective.

I hate the flaky tests just as much as a next dude but seriously, you need to factor in everything - deadlines, features, CI/CD, metrics etc, while still having constant time frame.

So lets not get into such polar opinions, its just not realistic at all. Life is never that ordered.

Re:Spark joy by running fewer tests by mpweiher on Saturday June 13, 2020 @04:57PM (#23512663)
If you came to me with that attitude, I would fire you immediately.

This is, per definition, critical functionality, and we can't figure out whether it is working or not, and you are saying you don't care.

Re:Spark joy by running fewer tests by maccard on Saturday June 13, 2020 @04:58PM (#23512675)
> If you came to me as developer telling me that you did 2 months debug on test that fails randmoly once in a week and you didn't do much else, I would probably show you the dor as a team lead. We need to put things into perspective.

This is a pretty polar opinion right here - it's not a flaky test that fails once a week, it's unreliable critical functionality. If I were _your_ lead and you told me you fired someone because they spent two months on something, I'd probably fire you for not knowing what your team are doing for two months.

Re:Spark joy by running fewer tests by viraptor on Sunday June 14, 2020 @12:17AM (#23515080)
> This is a pretty polar opinion right here - it's not a flaky test that fails once a week, it's unreliable critical functionality.

It may be unreliable critical functionality, or unreliable test. In some cases it's obvious which one it is, in some it isn't. If you know it's the test, it becomes much less beneficial to spend days on it.

Re:Spark joy by running fewer tests by majkinetor on Monday June 15, 2020 @04:23PM (#23532522)
I didn't say fire, didn't I, I said "show you the dor", like, get out from my office :P

And seriously, if you fire someone for having a strong opinion, you should remind yourself that even little kids tollerate a far worse behavior.

Re:Spark joy by running fewer tests by pavel_lishin on Friday June 12, 2020 @07:11AM (#23498070)
If it's acceptable for your critical functionality to fail 0.1% of the time in production, then sure, leave it in.
Re:Spark joy by running fewer tests by majkinetor on Saturday June 13, 2020 @08:07AM (#23508835)
Meh.

Majority of time, in my experience, flaky tests has nothing to do with the software but it is about faulty test infrastructure.

Re:Spark joy by running fewer tests by viraptor on Friday June 12, 2020 @10:18PM (#23506153)
There's difference between a flakey test and test failing due to randomly failing feature. They're not always the same thing. The test failing does not necessarily mean the feature doesn't work.