Getting anything done in such environments are just a pain. You spend more time fighting the systems than you do actually solving problems. It is my opinion that a CI/CD system needs just the following features: Triggers (source code repo, http endpoints or manually triggered), secret management and shell script execution. That's it, you can build anything using that.
If no one else is doing anything about the mess, then it falls to the senior person to sort it out.
As a rule of thumb:
- Ideally your people do the Right Thing by themselves by the magic of 'leadership'. - Second best: you chase the people to do the Right Thing. - Third: you as the senior person do the Right Thing. - Least ideal: no one fixes the mess nor implements the proper way.
I guess some people can achieve the ideal outcome with pure charisma (or fear?) alone, but I find that occasionally getting your hands dirty (option 3) helps earn the respect to make the 'leadership' work. It can also help ground you in the reality of the day to day work.
However, you are right that a senior person shouldn't get bogged down with such work. You need to cut your losses at some point.
* TC offers sh, full stop. If you want to script something that depends on bash, it's a PITA and you end up with a kludge to run bash in sh in docker in docker.
https://www.jetbrains.com/help/teamcity/command-line.html#:~...
Your "docker in docker" comment makes me wonder if you're conflating the image that you are using, that just happens to be run by TeamCity, versus some inherent limitation of TC. I believe a boatload of the Hashicorp images ship with dash, or busybox or worse, and practically anything named "slim" or "alpine" similarly
The trick is to NOT have your script depend on the specifics of the environment, but reverse the dependency. So replace all `If CI then Run X else if Local Run Y` with the ability to configure the script to run X or Y, then let the CI configure X and local configure Y. For example.
I'm not saying it is always easy and obvious. For bigger builds, you often really want caching and have shitloads of secrets and configurations going on. You want to only build what is needed, so you need something like a DAG. It can get complex fast. The trick is making it only as complex as it needs be, and only as reusable as and when it is actually re-used.
> I'm not saying it is always easy and obvious. For bigger builds, you often really want caching and have shitloads of secrets and configurations going on.
Here's the thing. When you don't want secrets, or caching, or messaging, or conditional deploys, none of this matters. Your build can be make/mvn/go/cargo and it just works, and is easy. It only gets messy when you want to do "detect changes since last run and run tests on those components", or "don't build the moon and the stars, pull that dependency in CI, as local users have it built already and it won't change." And the way to handle those situations involves running different targets/scripts/whatever and not what is actually in your CI environment.
I've lost count of how many deploys have been marked as failed in my career because the shell script for posting updates to slack has an error, and that's not used in the local runs of CI.
What you _actually_ need is a staging branch of your CI and a dry-run flag on all of your state modifying commands. Then, none of this matters.
"But then we can reuse the same pipeline for all our projects!"
The few bits that end up getting reused are the externalized bash scripts.
For me it was an epiphany as software dev - not to write reusable extensible scripts - I am so much more productive after that.
oh god just reading that gave me PTSD flash backs.
At $priorGig there was the "omni-chart". It was a helm chart that was so complex it needed to be wrapped in terraform and used composable terraform modules w/ user var overrides as needed.
Debugging anything about it meant clearing your calendar for the day and probably the following day, too.
and it feels like fighting against the flow when you're trying to make it reusable across many repos
Less snarky: https://learn.microsoft.com/en-us/virtualization/windowscont... seems to be a thing?
And in any case, you can use VMs instead of containers.
I've had the unfortunate pleasure of working with Windows Containers in the past. They are Containers in the sense that you can write a dockerfile for them, and they're somewhat isolated, but they're no better than a VM in my experience.
> And in any case, you can use VMs instead of containers.
They're not the same thing. If that's the case running linux AMI's on EC2 should be the same as containers.
It doesn’t really matter if you have to spin up a Linux instances and then run your build environment as a container there vs spinning up a windows VM.
1. Image tags ("latest", etc.) can change over time. Does any layer in your Dockerfile -- including inside transitive deps -- build on an existing layer identified by tag? If so, you never had reproducibility.
2. Plenty of Dockerfiles include things like "apt-get some-tool" or its moral equivalent, which will pull down whatever is the latest version of that tool.
It's currently common and considered normal to use these "features". Until that changes, Docker mostly adds only the impression of reproducibility, but genuine weight and pain.
Sure, if I try to rebuild a docker image from 5 years ago, it may fail, or produce something different, because it was pulling in some packages that have changed significantly in apt, or perhaps pip has changed encryption and no longer accepts TLS1.1 or whatever. And if you use ':latest' or if your team has a habit of reusing build numbers or custom tags, you may break the assumptions even sooner.
But even so, especially if using always incremented build numbers as image tags, a docker image will work the same way om the Jenkins system, the GitHub Actions pipeline, and every coworker's local build, for a while.
But really containerizing the build environment is not related to deterministic builds as there’s a lot more work needed to guarantee that. Including possible changes in the application itself.
Lastly you don’t need to use docker or dockerfiles to build containers. You can use whatever tooling you want to create a rootfs and then create an image out of that. A nifty way of actually guaranteeing reproducibility is to use nix to build a container image.
What would you say the goal of containerising builds is, if not reproducibility?
If you use a container with the same apt-get commands and the same OS, you already separate out almost all the reproducibility issues. What you get will be a very similar environment.
But it's not deterministic as you point out, that takes a lot more effort.
You can use specific image hashes to work around image tag changes.
For problem 2 ideally use something like NixOS as a base or at least nix package manager.
But IMO Nix is quite complicated, even though the working model is really good. So this is mostly if you really need deterministic builds.
Docker already gets you 80% of the way with 20% of the effort.
Also in many projects there’s things other than code that need to be “built” (assets, textures, translations, etc). Adding custom build targets to maven’s build.xml is truly not ideal then there’s people who actually try to write logic in there. That’s objectively worse than the YAML hell we were complaining about at the top of the thread.
Like I said, any version from the last, like, 10+ years (Java and Maven are both serious about backward compatibility), and if you install an ancient version you at least get fail-fast with a reasonable error.
> we really should be using gradle by now
We really shouldn't.
> Adding custom build targets to maven’s build.xml is truly not ideal then there’s people who actually try to write logic in there.
Maven doesn't have a build.xml, are you thinking of ant? With maven you write your custom build steps as build plugins, and they're written in plain old Java (or Kotlin, or Scala, or...) code, as plain old Maven modules, with the same kind of ordinary unit testing as your regular code; all your usual code standards apply (e.g. if you want to check your test coverage, you do it the same way as for your regular code - indeed, probably the same configuration you set up for your regular code is already getting applied). That's a lot better than YAML.
What in particular did you find "bad" with Python tech stack?
(I've got my gripes with Python and the tooling, but it's not this — I've got bigger gripes with containers ;-))
Stateful virtualenvs with no way to check if they're clean (or undo mistakes), no locking of version resolution (much less deterministic resolution), only one-way pip freeze that only works for leaf projects (and poorly even then), no consistency/standards about how the project management works or even basic things like the directory layout, no structured unit tests, no way to manage any of this stuff because all the python tooling is written in python so it needs a python environment to run so even if you try to isolate pieces you always have bootstrap problems... and most frustrating of all, a community that's ok with all this and tries to gaslight you that the problems aren't actually problems.
With docker containers, you can shell into it, do a couple of changes and "docker commit" it afterwards: similarly stateful, right? You resolve both by recreating them from scratch (and you could easily chmod -w the entire virtualenv directory if you don't want it to change accidentally).
The pattern of using requirements.txt.in and pip-freeze generated requirements.txt has been around for a looong time, so it sounds like non-idiomatic way to use pip if you've got problems with locking of versions or non-leaf projects.
As for directory layout, it's pretty clear it's guided by Python import rules: those are tricky, but once you figure them out, you know what you can and should do.
Can you clarify what do you mean with "structured unit tests"? Python does not really limit you in how you organize them, so I am really curious.
Sure, a bootstrapping problem does exist, but rarely do you need exactly a particular version of Python and any of the dev tools to be able to get a virtualenv off the ground, after which you can easily control all the deps in them (again a requirements-dev.txt.in + requirements-dev.txt pattern will help you).
And there's a bunch of new dev tools springing up recently that are written in Rust for Python, so even that points at a community that constantly works to improve the situation.
I am sorry that you see this as "gaslighting" instead of an opportunity to learn why someone did not have the same negative experience.
I guess theoretically you could, but I don't think that's part of anyone's normal workflow. Whereas it's extremely easy to run "pip install" from project A's directory with project B's virtualenv active (or vice versa). You might not even notice you've done it.
> You resolve both by recreating them from scratch
But with Docker you can wipe the container and start again from the image, which is fixed. You don't have to re-run the Dockerfile and potentially end up with different versions of everything, which is what you have to do with virtualenv (you run pip install and get something completely different from the virtualenv you deleted).
> you could easily chmod -w the entire virtualenv directory if you don't want it to change accidentally
But you have to undo it every time you want to add or update a dependency. In other ecosystems it's easy to keep my dependencies in line with what's in the equivalent of requirements.txt, but hard to install some random other unmanaged dependency. In the best ecosystems there's no need to "install" your dependencies at all, you just always have exactly the packages listed in the requirements.txt equivalent available at runtime when you run things.
> The pattern of using requirements.txt.in and pip-freeze generated requirements.txt has been around for a looong time, so it sounds like non-idiomatic way to use pip if you've got problems with locking of versions or non-leaf projects.
I've literally never seen a project that does that. And even if you do that, it's still harder to work with because you can't upgrade one dependency without unlocking all of your dependencies, right?
> As for directory layout, it's pretty clear it's guided by Python import rules
I don't mean within my actual code, I mean like: where does source code go, where does test code go, where do non-code assets go.
> Can you clarify what do you mean with "structured unit tests"?
I mean, like, if I'm at looking at a particular module in the source code, where do I go to find the tests for that module? Where's the test-support code as distinct from the specific tests?
> rarely do you need exactly a particular version of Python and any of the dev tools to be able to get a virtualenv off the ground
Whether virtualenv is available is a relatively recent change, so you already have a fractal problem. Having an uncontrolled way of installing your build environment is another of those things that's fine until it isn't.
> And there's a bunch of new dev tools springing up recently that are written in Rust for Python
Yeah, that's the one thing that gives me some hope that there might be light at the end of the tunnel, since I hear they mostly ignore all this idiocy (and avoid e.g. having user-facing virtualenvs at all) and just do the right thing. Hopefully once they catch on we'll see Python start to be ok without containers too and maybe the container hype will die down. But it's certainly not the case that everything has been fine since 2007; quite the opposite.
I've rarely seen a feedback loop with containers that's not longer than 10s only due to containerization itself, and that breaks the "golden" 10s rule (see https://www.nngroup.com/articles/response-times-3-important-...).
If you aim for quicker turn-around (eg. just running a single test in <1s), you'll have to either aggressively optimize containers (which is pretty non-idiomatic with Docker containers in particular), or do away with them.
Why would it be slow?
It needs to be rebuilt? (on a fast moving project with mid-sized or large team, you'll get dependency or Dockerfile changes frequently)
It needs to restart a bunch of dependent services?
Container itself is slow to initialize?
Caching of Docker layers is tricky, silly (you re-arrange a single command line and poof, it's invalidated, including all the layers after) and hard to make the most of.
If you can't get a single test running in <1s, you are never going to get a full test suite running in a couple of seconds, and never be able to do an urgent deploy in <30s.
Sounds like a skill issue tbh.
`time podman run —-rm -it fedora:latest echo hello` will return in a few milliseconds, whatever delay you are complaining about would be from the application running in the container.
Lastly containers != docker.
Or write your stuff in eg Python in the first place. GitHub's CI yaml supports scripts in arbitrary languages, not just shell.
This syndrome is very common these days. Things are explained differentially: it's like Circle CI but in the GH repo. Well that's no use if the audience wasn't around when Circle CI was first new and readily explained (It's like Jenkins but in the cloud...).
CI/CD is a method to automate tasks in the background that you would otherwise run on your laptop. The output
of the tasks are used as quality gates for merging commits, and for deployments.
- Step 1. Your "laptop in the cloud" requires some configuration (credentials, installed software, cached artifacts)
before a job can be run.
- Requires logic specific to the CI/CD system
- Step 2. Running many jobs in parallel, passing data from step to step, etc requires some instructions.
- Requires logic specific to the CI/CD system
- Step 3. The job itself is the execution of a program (or programs), with some inputs and outputs.
- Works the same on any computer (assuming the same software, environment, inputs, etc)
- Using a container in Step 1. makes this practical and easy
- Step 4. After the job finishes, artifacts need to be saved, results collected, and notifications sent.
- Some steps are specific to the CI/CD system, others can be a reusable job
Step 3 does not require being hard-coded into the config format of the CI/CD system. If it is instead
just executable code in the repo, it allows developers to use (and work on) the code locally without
the CI/CD system being involved. It also allows moving to a different CI/CD system without ever rewriting
all the jobs; the only thing that needs to be rewritten are the CI/CD-specific parts, which should be
generic and apply to all jobs pretty much the same.
Moving the CI/CD-specific parts to a central library of configuration allows you to write some code
once and reuse it many times (making it DRY). CircleCI Orbs, GitHub Actions, Jenkins Shared Libraries/
Groovy Libraries, etc are examples of these. Write your code once, fix a bug once, reuse it everywhere.Most programmers have never written a shell script and writing CI files is already frustrating because sometimes you have to deploy, run, fix, deploy, run, fix, which means nobody is going to stop in the middle of that and try to learn shell scripting.
Instead, they copy commands from their terminal into the file and the CI runner takes care of all the rough edges.
I ALWAYS advise writing a shell script but I know it's because I actually know how to write them. But I guess that's why some people are paid more big bux.
That's generally a bit less convenient, ie it takes a few more lines, but it has significantly fewer sharp edges than your typical shell script. And more people have written Python scripts, I guess?
Python’s subprocess has communicate(), check_output() and other helpers which takes care of a lot but (a) you need to know what method you should actually call and (b) if you need to do something outside of that, you have to use Popen directly and it’s much more work than just writing a shell script. All doable if you understand pipes and all that but if you don’t, you’ll be just throwing stuff at the wall until something sticks.
Often the shell script just calls a python/Ruby/rust exec anyways...
Shell scripts are for scripting...the shell. Nothing else.
Your average person is going to be blindsighted.
The 'Frankenstein' approach isn't what makes it necessarily worse. Eg Makefiles work like that, too, and while I have my reservations about Make, it's not really because they embed shell scripts.
I've struggled in some teams to explained why it's better to extract your command in scripts (ShellCheck on it, scripts are simple to run locally etc...) instead of writing a Frankenstein of YAML and shell commands. I hope someday to find an authoritative guidelines on writing pipeline that promote this approach so at least I can point to this link instead of defending myself being a dinosaur!