News for hackers. Stuff that matters.
Re:Meta developer tools: Working at scale by erik_seaberg on Thursday June 29, 2023 @01:02AM (#36516504)
Maybe I misinterpreted “flaky test detection, suppression” and “not going to fail builds.” You retry until a confirmed success before merging?
Re:Meta developer tools: Working at scale by vlovich123 on Thursday June 29, 2023 @02:18AM (#36517006)
I think flaky tests would be always retried and if they start failing 100% then it’s marked as a pure regression. But they don’t block merges if they fail (ie it might be done only after the merge). Not 100%. But generally yes - the culture is to incentivize people to fix their flaky tests and it’s one of the things EMs are judged on.

> Ultimately, our goal with PFS is not to assert that any test is 100 percent reliable, because that’s not realistic. Our goal is to simply assert that a test is sufficiently reliable and provide a scale to illustrate which tests are less reliable than they should be.