Re:We cut our CI pipeline execution time in half
by mikepurvis on Friday February 17, 2023 @12:32PM
(#34837615)
We do the pre-baked dependency images too, and it's definitely workable, but I feel like it's a lot of overhead maintaining those— you have to build and distribute and lifecycle them, and it's extra jobs to monitor. I wish tools like GitLab CI had more automated workflows for being able to automatically manage those intermediate containers, eg:
- Here's a setup stage, the resulting state of which is to be saved as a container image and used as the starting point for any follow-on stages that declare `image: <project/job/whateever>`
- Various circumstances should trigger me to be rebuilt: weekly on Saturday night, whenever repo X has a new tag created, whenever I'm manually run, whenever a special parallel "check" stage takes more than X minutes to complete, etc.
Ultimately, I think the necessity for all this kind of thing really just exposes how weak the layered container image model is— something like Nixery that can delivery ad-hoc environments in a truly composable way is ultimately a much better fit for this type of use-case, but it has its own issues with maturity.
- Here's a setup stage, the resulting state of which is to be saved as a container image and used as the starting point for any follow-on stages that declare `image: <project/job/whateever>`
- Various circumstances should trigger me to be rebuilt: weekly on Saturday night, whenever repo X has a new tag created, whenever I'm manually run, whenever a special parallel "check" stage takes more than X minutes to complete, etc.
Ultimately, I think the necessity for all this kind of thing really just exposes how weak the layered container image model is— something like Nixery that can delivery ad-hoc environments in a truly composable way is ultimately a much better fit for this type of use-case, but it has its own issues with maturity.