Re:The End of CI
by hinkley on Sunday July 03, 2022 @02:36PM
(#31969918)
It’s the same problem with tests in general. Developers think it’s all about getting a green build. It’s not. It’s about fixing red builds as fast as possible. I have a whole other rant about making zero build failures a metric.
A red local build probably means a red CI build, but a red CI build doesn’t necessarily mean a red local build. Now you’re fucked because there’s a build failure you can’t reproduce.
Reducing variance helps. Having the person who set up the CI system also copy edit the onboarding docs helps a lot with this.
It’s a matter of scale. As the team grows and particularly as you hit the steep part of the S curve of development, you’re going to have lots of builds and that 1:50 error is going to go from every two weeks to once a day.
Humans interpret every day as “all the time”. Some do this for every week, especially if it coincides with their most important commits. It’s not the ratio of failures that bothers people. It’s the frequency, and the clusters.
A red local build probably means a red CI build, but a red CI build doesn’t necessarily mean a red local build. Now you’re fucked because there’s a build failure you can’t reproduce.
Reducing variance helps. Having the person who set up the CI system also copy edit the onboarding docs helps a lot with this.
It’s a matter of scale. As the team grows and particularly as you hit the steep part of the S curve of development, you’re going to have lots of builds and that 1:50 error is going to go from every two weeks to once a day.
Humans interpret every day as “all the time”. Some do this for every week, especially if it coincides with their most important commits. It’s not the ratio of failures that bothers people. It’s the frequency, and the clusters.