News for hackers. Stuff that matters.
Re:Dagger: a new way to build CI/CD pipelines by drewm1980 on Thursday March 31, 2022 @01:47AM (#30864199)
The carbon footprint of the cloud has exceeded the footprint of air travel. A move away from monolithic statically compiled binaries to constellations of microservices (usually bloated docker containers) is a significant part of the problem.

Docker's explosive growth is partly due to the convenience of the abstraction it provides, abstracting the entire linux userspace, putting even OS-wide package managers and language-specific package managers inside another box. This usually breaks any caching / code sharing that the now containerized packages had, resulting in the bloat. The docker image is portable, yes, but disk and RAM efficiency of the systems people are building are awful. It has been the norm for every little microservice to add a few GB of bloat to the overall software system. A dev writes "RUN pip install pytorch" and you have CICD servers pulling down 2GB of pytorch to build the container, every time the software is built, probably forever. Meanwhile species are going extinct and a lot of people are starting to wonder if it's ethical to work in technology at all.

What can your team do to reverse this tragedy of the commons? Can you come up with some equally ergonomic tool that can migrate the container ecosystem on to something that has a solid foundation with good caching?

Re:Dagger: a new way to build CI/CD pipelines by cakoose on Thursday March 31, 2022 @03:32AM (#30864748)
> What can your team do to reverse this tragedy of the commons?

It's tragedies all the way down :-)

Making a successful tool in a competitive space is hard enough. Asking a creator to somehow factor in environmental impact isn't going to work. A creator that places additional constraints on themselves will more likely lose out to a competitor that doesn't.

This is the kind of thing a carbon tax is perfect for. Cost optimization is infused into how businesses and individuals operate. Tax the things you care about and watch all that machinery get to work!

Right now, people use Docker (and dynamic languages, bloated JS libraries, etc) because burning electricity is the cheapest route. (Sort of... the market's obviously not perfect.) Make electricity expensive enough and we'll get more efficient systems.

And of course taxes aren't the only solution. For example, Google Flights now shows the carbon cost of each flight option, which I think might actually move the needle. But that category of solution might only work in an industry with similar characteristics, e.g. infrequent but large purchases, mature industry that can define measurement standards.

Re:Dagger: a new way to build CI/CD pipelines by ubercore on Thursday March 31, 2022 @05:58AM (#30865484)
> Asking a creator to somehow factor in environmental impact isn't going to work.

That attitude doesn't leave a great impression for me. I take your points about how difficult it is, but I think we can all do better than throw our hands up in the air. For instance, you could talk about how easy containerization of CI/CD makes it easier to move your pipeline where impact is lowest. Or that you can control your own impact rather than leave it up to the whim of someone like CircleCI.

There's no silver bullet with environmental impact, which is why we all have to collectively apply whatever wins we can, wherever we can.

Re:Dagger: a new way to build CI/CD pipelines by xeromal on Thursday March 31, 2022 @10:53AM (#30867865)
I haven't fully formed this thought yet, but your request might result in a net loss of "good" people. The people who actually consider your request will probably fail to launch their business due to those environmental constraints. If you take 10 people who run a business regular and compare them to 10 who do it in an environmentally conscience way (while environmentalism not being he the bizmodel in the first place), I'd think you'd have equal or less environmental businesses succeeded which actually ends up hurting your problem more than helping.

I think the ideal way is to let a startup thrive any way possible and when they're not trying to not starve anymore, begin environmental changes.

Re:Dagger: a new way to build CI/CD pipelines by cakoose on Thursday March 31, 2022 @06:02PM (#30872696)
> For instance, you could talk about how easy containerization of CI/CD makes it easier to move your pipeline where impact is lowest. Or that you can control your own impact rather than leave it up to the whim of someone like CircleCI.

Oh yeah -- I'm not at all saying Docker is all bad.

* Like you mentioned, increasing interoperability allows the market be more efficient.

* Docker continued the path that VMs started towards making strong isolation even more efficient and accessible.

* Layers and caching are obviously good for resource consumption.

It's just that the original comment seemed to try shaming the Docker creators about what they've built. All they did was try and make something better. And if they didn't, someone else would have.

Re:Dagger: a new way to build CI/CD pipelines by drewm1980 on Thursday April 07, 2022 @06:14AM (#30942661)
It was meant as genuine constructive criticism and a plea to do better. They have more traction than most people to morph docker into something that comes close to its predecessors in efficiency. Somehow replace their fragile layer caching strategy based around diffs of the entire filesystem, with something that understands the OS and language-specific packaging systems being used inside the containers, and can therefore at ~least cache the package downloads. We desperately need a better, more efficient package manager, and Docker has been a huge setback; it has become normal for CICD to rebuild your image every time you touch the code, and pull down practically an entire linux distribution every time. I know you can do better with docker caching if you put in enough consistent and dedicated engineering effort into how you structure things, and set up local apt and pypi mirrors, etc... but that's the default behavior, and none of (admittedly very small number organizations) I've worked at have had the organizational capacity to really get past it. I don't know if we need what they're building, but we absolutely need a much more efficient new version of Docker with an easy migration path. Indeed they may be the ~only people with traction to resolve this, since in the current climate as soon as someone introduces a new package manager with more efficient dependency resolution or caching (i.e. nix, poetry, cargo...), people just stick it in their Docker container and break the caching anyway.
Re:Dagger: a new way to build CI/CD pipelines by dradtke on Thursday March 31, 2022 @02:23PM (#30870579)
One of the big points Bill Gates makes in his recent book on climate change is that once the genie is out of the bottle in terms of lifestyle, there's no going back. It's naive to expect people to willingly reduce their energy use enough to make an impact, and immoral when you consider all the people in poor villages that don't even have electricity yet.

The solution is, in a nutshell, to electrify everything, and push to make electricity clean and plentiful. Anything else is doomed to fail because we can't beat climate change by reducing our carbon footprint; we have to eliminate it entirely.

From that perspective, cloud energy usage is not a problem, since it's already electrified by nature. Now we just need to stop emitting carbon in order to make electricity (among other things).

Re:Dagger: a new way to build CI/CD pipelines by scg on Thursday March 31, 2022 @04:15AM (#30864971)
> a lot of people are starting to wonder if it's ethical to work in technology at all

Can you share some evidence to support this claim? Who thinks we're better off addressing climate change, etc. with less technology?

Re:Dagger: a new way to build CI/CD pipelines by austinshea on Tuesday April 05, 2022 @08:45AM (#30918265)
“A move away from monolithic statically compiled binaries to constellations of microservices (usually bloated docker containers) is a significant part of the problem.”

Untrue, in every way.

Why did you say this?

Re:Dagger: a new way to build CI/CD pipelines by drewm1980 on Thursday April 07, 2022 @07:01AM (#30942935)
It won't be true in every shop, but I do this professionally and it's been my firsthand experience. A native statically compiled binary containing just the functions that actually get called will usually be... 10-100 MB. Ungroomed Docker images are ~10GB-20GB, same as you'd have on the root partition if you sat down and brought up a linux workstation or server node manually, and this is not a coincidence. Sure, docker avoids duplicating the linux kernel, making it more efficient than an old school VM, but these days all the ~other software bloat dominates the kernel in size. Most companies do not have a 100 person team of engineers dedicated to optimizing their image build and management workflow, and pruning what goes into their containers.