News for hackers. Stuff that matters.

A comment on Meta developer tools: Working at scale. Parent comment

Re:Meta developer tools: Working at scale by vlovich123 on Wednesday June 28, 2023 @08:23PM (#36514695)
It determines flakiness in aggregate across all builds. It also does automatic retries but it does so for the purposes of marking a run as flaky which it then remembers. And it's your team's responsibility for making your tests stable. If they're not stable, they're not going to fail builds which is the correct response (a ticket is filed & it's your team's job to keep those tickets under control). The neat thing is also that if you fix the test and it's no longer flaky, it gets automatically added back in & the ticket is closed (in case you forgot).

https://engineering.fb.com/2020/12/10/developer-tools/probab...

[ Parent | on HN ]
Re:Meta developer tools: Working at scale by erik_seaberg on Wednesday June 28, 2023 @10:17PM (#36515551)
If a test goes from “flaky” to “never succeeds,” I absolutely want to fail builds. To me it seems pretty common for large integration tests to be flaky, and I don’t know that it’s always feasible to avoid without mocking so much that the test becomes meaningless.
[ Parent | on HN ]
Re:Meta developer tools: Working at scale by vlovich123 on Thursday June 29, 2023 @12:37AM (#36516335)
What makes you think that’s not the case?
[ Parent | on HN ]
Re:Meta developer tools: Working at scale by erik_seaberg on Thursday June 29, 2023 @01:02AM (#36516504)
Maybe I misinterpreted “flaky test detection, suppression” and “not going to fail builds.” You retry until a confirmed success before merging?
[ Parent | on HN ]
Re:Meta developer tools: Working at scale by vlovich123 on Thursday June 29, 2023 @02:18AM (#36517006)
I think flaky tests would be always retried and if they start failing 100% then it’s marked as a pure regression. But they don’t block merges if they fail (ie it might be done only after the merge). Not 100%. But generally yes - the culture is to incentivize people to fix their flaky tests and it’s one of the things EMs are judged on.

> Ultimately, our goal with PFS is not to assert that any test is 100 percent reliable, because that’s not realistic. Our goal is to simply assert that a test is sufficiently reliable and provide a scale to illustrate which tests are less reliable than they should be.

[ Parent | on HN ]