News for hackers. Stuff that matters.
Re:We're Leaving Kubernetes by pmarreck on Monday November 04, 2024 @03:19PM (#42045620)
In my last role as a director of engineering at a startup, I found that a project `flake.nix` file (coupled with simply asking people to use https://determinate.systems/posts/determinate-nix-installer/ to install Nix) led to the fastest "new-hire-to-able-to-contribute" time of anything I've seen.

Unfortunately, after a few hires (hand-picked by me), this is what happened:

1) People didn't want to learn Nix, neither did they want to ask me how to make something work with Nix, neither did they tell me they didn't want to learn Nix. In essence, I told them to set the project up with it, which they'd do (and which would be successful, at least initially), but forgot that I also had to sell them on it. In one case, a developer spent all weekend (of HIS time) uninstalling Nix and making things work using the "usual crap" (as I would call it), all because of an issue I could have fixed in probably 5 minutes if he had just reached out to me (which he did not, to my chagrin). The first time I heard them comment their true feelings on it was when I pushed back regarding this because I would have gladly helped... I've mentioned this on various Slacks to get feedback and people have basically said "you either insist on it and say it's the only supported developer-environment-defining framework, or you will lose control over it" /shrug

2) Developers really like to have control over their own machines (but I failed to assume they'd also want this control over the project dependencies, since, after all, I was the one who decided to control mine with the flake.nix in the first place!)

3) At a startup, execution is everything and time is possibly too short (especially if you have kids) to learn new things that aren't simple, even if better... that unfortunately may include Nix.

4) Nix would also be perfect for deployments... except that there is no (to my knowledge) general-purpose, broadly-accepted way to deploy via Nix, except to convert it to a Docker image and deploy that, which (almost) defeats most of the purpose of Nix.

I still believe in Nix but actually trying to use it to "perfectly control" a team's project dependencies (which I will insist it does do, pretty much, better than anything else) has been a mixed bag. And I will still insist that for every 5 minutes spent wrestling with Nix trying to get it to do what you need it to do, you are saving at least an order of magnitude more time spent debugging non-deterministic dependency issues that (as it turns out) were only "accidentally" working in the first place.

Re:We're Leaving Kubernetes by oblio on Monday November 04, 2024 @03:46PM (#42045825)
I think if you take about 80% of your comment and replace "Nix" with "Haskell/Lisp" and a few other techs, you'd basically have the same thing. Especially point #1.
Re:We're Leaving Kubernetes by strawhatguy on Monday November 04, 2024 @06:56PM (#42047357)
Too true. I think there's a lot of people who don't want control; freedom is responsibility, as the saying goes, and responsibility can be stressful, even if it's liberating also.

Worse is better, sadly.

Re:We're Leaving Kubernetes by rfoo on Monday November 04, 2024 @04:26PM (#42046147)
In a worse world, worse is better.
Re:We're Leaving Kubernetes by pmarreck on Tuesday November 05, 2024 @02:25AM (#42049368)
this is such an awful truth to even say aloud lol but... yeah.

which is why you gotta find your peeps that believe in the better world and are thus believing it into existence. (OT: reminds me of this song title: https://soundcloud.com/anjunabeats/mat-zo-see-it-when-i-beli...)

Re:We're Leaving Kubernetes by kalaksi on Monday November 04, 2024 @05:33PM (#42046700)
From my perspective, installing Nix seems pretty invasive. I can understand if someone doesn't want to mess with their system "unnecessarily" especially if the tool and it's workings are foreign. And I can't really remember the last time I had issues with non-deterministic dependencies either. Dependency versions are locked. Maybe I'm missing something?
Re:We're Leaving Kubernetes by fragmede on Monday November 04, 2024 @09:28PM (#42048092)
Typically when you start a new dev job the company will provide you with a pre-provisioned laptop that has their security stuff setup and maybe dev tools already installed, eg source code, compilers, VMs, Nix, and a supported editor, so it's not exactly a personal machine that they're messing with.
Re:We're Leaving Kubernetes by kalaksi on Tuesday November 05, 2024 @01:45AM (#42049151)
Sure, and personally, I have no issue installing and using recommended tools since it's indeed just a work laptop, but I've seen more tuned setups too (see parent's point 2: "Developers really like to have control over their own machines")
Re:We're Leaving Kubernetes by pxc on Tuesday November 05, 2024 @01:20AM (#42049025)
> From my perspective, installing Nix seems pretty invasive.

How so? With what other software does Nix interfere?

Re:We're Leaving Kubernetes by kalaksi on Tuesday November 05, 2024 @01:35AM (#42049100)
From the provided link:

> Nix requires a broad set of changes to your system, from creating new users to installing and running a daemon to creating a root volume and beyond

Re:We're Leaving Kubernetes by pmarreck on Tuesday November 05, 2024 @02:03AM (#42049238)
The daemon and the users are only to run a build when that is necessary. It might seem invasive but it works out pretty well.
Re:We're Leaving Kubernetes by pxc on Tuesday November 05, 2024 @03:18AM (#42049622)
Ok, I see what you mean now.

What the post is trying to do there is motivate the creation of a new installer, including to the existing Nix community. The snippet you've highlighted is essentially correct, but I still wouldn't characterize Nix as particularly invasive.

The only that Nix strictly needs is to be plugged into your shell. That's it. It doesn't need deep or special hooks into a system just to function.

But including the daemon enables sandboxing for builds that Nix performs, which improves both the security and isolation of those builds, and it also lets Nix be shared nicely between unprivileged users on multiuser systems. For those reasons, daemonful installs are the default and with them come the system users.

(Adding system users is pretty much bog standard stuff for Unix system software, since the main kind of security boundary designed into that system is boundaries between users. Indeed, that's exactly what that's used for with Nix, too.)

The two things I described above comprise the totality of what is required to enable all of Nix's functionality. Everything else that the Determinate Nix installer does as of now is to work around or avoid macOS quirks, and is totally unnecessary for using Nix on any other OS.

The 'root volume' stuff is the result of a collision between the historical and conventional location of the Nix store at `/nix` and Apple's later imposition of a read-only root partition. So Nix installers do a little Apple-specific dance that creates a kind of filesystem volume that doesn't take up any real space or involve any physical partitioning of the disk when they run on macOS.

The other thing this installer does is build in an attempt to self-repair the damage that Apple inflicts upon Nix's sole real requirement by having macOS unconditionally clobber the shell config files under /etc during major macOS updates.

That's it. That's an exhaustive list of all the things a Nix installer does and why. It's not particularly tricky, or hard to remember or figure out. It's not even hard to undo manually— before the Determinate Nix installer existed, I sometimes uninstalled Nix by hand while manually testing the macOS bootstrap scripts for my dotfiles. It was annoying to do, and the uninstallation functionality of the Determinate Nix installer is extremely reliable and convenient and nice. But anyone who knows what `$PATH` is and has ever run `man` before could completely uninstall Nix even if some joker walked over to their machine and deleted the uninstaller.

At the same time, none of the changes Nix installers make on your system affect the behavior of outside programs at all, except by exposing what you choose to install via Nix through standard Unix environment variables like PATH.

Lacking things like kernel components, automatic self-updates, or the requirement for privileged APIs (e.g., on macOS, the endpoint security APIs and accessibility APIs), Nix is not only far less invasive than any endpoint security software, monitoring software, or MDM software you are likely to run on a work machine, but I'd argue tons of common desktop software like Zoom, Discord, DisplayLink and tons of popular macOS powertools like Amphetamine, SteerMouse, SoundSource, etc.

Plus the uninstall procedure with the DetSys installer and its forks is totally conventional and leaves nothing behind: run uninstaller, thing gone.

Nix on macOS is admittedly not an installer-free, drag-and-drop app bundle like some lovely applications get to be. But at most workplaces it's not likely to crack the top 10 most invasive applications installed on the average developer machine, either. Nix installers are just very up front about the things they do set up.

All that said, there are reasonable people who find having a daemon at all offensive. People who are deeply committed to minimalism or simplicity might prefer a single-user install or to use some other tool. But I think for most people, Nix is imo more than fine in terms of invasiveness.

Re:We're Leaving Kubernetes by pmarreck on Tuesday November 05, 2024 @01:53AM (#42049189)
> installing Nix seems pretty invasive

I'm writing a test to check whether a tool I'm writing can work without Nix (it works with it perfectly, but I want it to also work without it because there are a lot of folks like you, and like me about 3 years ago, who still think they'd rather struggle with manually installing the right glibc that goes with the right python dependency installed with the right pip and venv versions, to the right location, that goes with the right python version that makes Whisper models work (literally the thing I'm currently working on), instead of just running `nix develop` and getting a coffee and then done.

And all I have to do to simulate "no Nix" is to remove all the nix paths from PATH (I suppose I could purge it from the linker paths as well, now that I think about it). But that's it.

What Nix does is put its entire repo into a separate part of your hard drive owned by root, and create a few build users for security reasons. That's (to me) not particularly "invasive," but YMMV (and if you use the Determinate Nix installer, it's even more trivial to uninstall than the official way). Also, when you run `nix develop`, the environment changes it does to make everything "just work" (like PATH changes etc) are only valid for that terminal session. Again, this is the least intrusive thing possible while also providing the guarantees it does, and is also (more or less) guaranteed to work.

The Nix whitepaper is pretty readable and not that long. I recommend it to understand why it's important and useful: https://edolstra.github.io/pubs/nspfssd-lisa2004-final.pdf

There is also Guix, which is like Nix but uses Guile (a Scheme dialect) as its scripting language all the way down to the bare metal (literally, the boot loader is written in it, I believe, as soon as the interpreter is loaded somehow). Their strategy seems to be to let Nix take the lead and make all the mistakes and then implement the way that seems to work the best, in its own ecosystem/tooling: https://guix.gnu.org/ But they have a lot fewer packages than Nix does.

Both of these let you define an entire machine with a single configuration file that is far more guaranteed to work than running a Dockerfile.

Re:We're Leaving Kubernetes by herewulf on Tuesday November 05, 2024 @02:37AM (#42049410)
Guix uses GRUB like most distros (but is scripted a bit to allow booting earlier generations of the system). The init system is Guile (Shepherd).

Fewer packages, yes, but the packages are by far the most common ones. It's easy to add packages for yourself, if needed. Nonguix channel and others for stuff upstream won't accept.

I believe Guix is innovating on a number of things in relation to Nix these days. So I've heard. I don't know much about Nix, honestly.

I heartily agree with your last paragraph in the case of Guix.

Re:We're Leaving Kubernetes by kalaksi on Tuesday November 05, 2024 @03:10AM (#42049576)
Thanks for providing some concrete examples and explaining how it works. Sounds like a reasonable setup to automate. I'm actually running NixOS at home on a few hosts (but not on my desktop), so Nix isn't entirely foreign to me, but I haven't (yet?) used it with other distros or for developing as my needs have been simpler. In your case, I probably would have chosen the recommended path.

In my current work project, we use Windows and .NET with some libs and tools. Nothing too complicated, but automating it would be nice. I could probably push for it a bit, but I'm not familiar with automating Windows environments, since I mainly use Linux at home.

Re:We're Leaving Kubernetes by rtpg on Tuesday November 05, 2024 @05:59AM (#42050481)
Nix on MacOS makes a bunch of build users and has a bad habit of touching files that then get touched by MacOS updates, leading to a system that works... then doesn't work.
Re:We're Leaving Kubernetes by jen20 on Tuesday November 05, 2024 @08:07AM (#42051199)
The only such case I’ve seen was the early betas of Sequoia requiring an extra flag to use higher UIDs for build users.
Re:We're Leaving Kubernetes by pmarreck on Tuesday November 05, 2024 @11:44PM (#42057252)
The Determinate Nix installer flavor is supposedly more resilient to this. Have survived at least 1 OS update unscathed with it.
Re:We're Leaving Kubernetes by bamboozled on Monday November 04, 2024 @05:42PM (#42046800)
Try Devbox, you can basically ignore nix entirely and reap all the benefits.
Re:We're Leaving Kubernetes by pmarreck on Tuesday November 05, 2024 @02:28AM (#42049377)
When I looked at it, I encountered some problem that unfortunately slips my mind now.
Re:We're Leaving Kubernetes by bamboozled on Thursday November 07, 2024 @04:25AM (#42075095)
We have issues with it from time to time, it's under fairly active development, give it a try.
Re:We're Leaving Kubernetes by nixdev on Monday November 04, 2024 @07:23PM (#42047504)
Heh, yeah. You gotta put in writing that only userlands defined in Nix will be eligible to enter any environment beyond "dev". And (also put in writing) that their performance in the role will be partly evaluated on their ability to reach out for help with Nix when they need it.
Re:We're Leaving Kubernetes by sgarland on Monday November 04, 2024 @07:26PM (#42047526)
People just straight-up don’t want to learn. There are always exceptions, of course, but IME the majority of people in tech are incurious. They want to do their job, and get paid. Reading man pages is sadly not in that list.
Re:We're Leaving Kubernetes by jrockway on Monday November 04, 2024 @08:14PM (#42047751)
This has always made me sad, but I think you're right in a lot of cases. What I've always tried to do is to focus on basic productivity; make sure everyone has everything they need to do their work, and that most people do it in the same way, so you can make progress on the learning journey together. Whenever people ask me for help and want to set up a meeting (not just "please answer this on Slack and I'll leave you alone"), I record the meeting, try to touch on all the related areas of their problem, and then review the recording for things that would be interesting to write about it. If any of the digressions are interesting, I go into Notion, create a new page, and write up a couple paragraphs. Then I give my team "ever wonder what dynamic linking is and how to debug it?" and they can read it and know as much as I know.

I really, really struggle to deal with the fact that people don't know as much as I do (I wrote my first program when I was 4 and I'm 39 now), but I have accepted that it's not a weakness on their part, it's a weakness on my part. I wouldn't lower my standards (as a manager once suggested), but I do feel like it's my obligation to lead them on a journey of learning. That is to say, people don't learn without teaching, so be a teacher.

Re:We're Leaving Kubernetes by dilyevsky on Monday November 04, 2024 @10:16PM (#42048284)
> They want to do their job, and get paid

Where "job" is defined in a narrowest way possible to assume minimum responsibility. Still want to get 200k+ salaries though...

This may sound extreme (it really isn't) but as Dr of Eng TP's job was to sus those folks out as early as possible and part ways (the kind where they go work for someone else). Some folks are completely irrational about their setups and no amount of appeasement in the form of "whys" and training is usually sufficient.

Re:We're Leaving Kubernetes by chipdart on Tuesday November 05, 2024 @02:26AM (#42049372)
> People just straight-up don’t want to learn. (...) but IME the majority of people in tech are incurious. They want to do their job, and get paid. Reading man pages is sadly not in that list.

I don't think you know what you're talking about. Just because you know people who do not want to waste their time on a set of unproductive chores you arbitrarily singled out, that does not mean they are against learning.

Your take is particularly absurd considering the topic: engineers working on distributed services.

Do you actually believe that you build up enough knowledge on this topic to become a professional in the field if you "straight-up don't want to learn"? There is not a single developer in the field who, at least to some degree, is not self-taught.

> They want to do their job, and get paid.

Everyone wants to get paid. Do you know anyone who works non-profit?

What you're failing to understand is the "do their job" part. Software developers are trained to solve the problems they face, and not waste time with the problems they do not have. Time is precious, and they invest it where it has the largest return on investment.

> Reading man pages is sadly not in that list.

Man pages are notoriously a colossal waste of time. In general they are poorly thought out, they are incomplete, they were written with complete disregard for user experience, and more often than not they are way out of date.

Why do you think sites like Stack overflow is so popular? Because all those "incurious" people in tech feels the need to ask questions and dig through answers on how to solve problems?

I think you're just picking a very personal definition of competence which conveniently boils down to "do the things I do, and do not do the things I don't". Except the bulk of the people in the field is smart, and some have already solved problems that you aren't aware exist, such as wasting precious time deciphering unreadable documents that are systematically out of date.

Re:We're Leaving Kubernetes by pxc on Tuesday November 05, 2024 @04:40AM (#42050062)
> Man pages are notoriously a colossal waste of time. In general they are poorly thought out, they are incomplete, they were written with complete disregard for user experience, and more often than not they are way out of date.

Uh, what? What man pages are you reading? I read manpages all the time, and I've never run into an issue where one contained info that was untrue because outdated. The only manpages I've ever read that I'd characterize as incomplete are Apple's.¹

> Why do you think sites like Stack overflow is so popular? Because all those "incurious" people in tech feels the need to ask questions and dig through answers on how to solve problems?

One of the reasons Stack Overflow is so popular is that people who can't/won't read docs can use it to have answers spoonfed to them, often by people who only differ from them in being more willing/able to read the docs. Isn't that extremely obvious?

> unreadable documents

Reading isn't a singular skill— each genre requires its own skills, and you gradually pick those up by reading in that genre. Reading novels doesn't much prepare you to read math textbooks, but that doesn't make all math textbooks 'unreadable'.

The same things goes for skimming. Skimming a text is likewise a (set of) genre-specific skill(s), built up through practice.

Frankly, moving from your terminal to your web browser to look up how to use a CLI tool is only consistently faster than working with the docs native to that CLI environment (man pages, info pages, usage messages, --help flags, help subcommands, tldr pages, etc.) if you have don't have very good reading skills in the genres of those native docs.

As someone who does not have difficulty skimming or navigating manpages quickly, when someone tells me that digging through StackOverflow seems like less of a waste of time than reading docs and so they never read docs, I have to wonder if the real issue is that a reading skills deficit is caught in a self-reinforcing loop.

And indeed, a trip to StackOverflow never ends at StackOverflow for a person with much curiosity. Because even if a curious person finds a solution to their immediate problem, they will wonder things like:

  - is this solution outmoded by some other fix?
  - how is the feature/option/change used in this solution actually supposed to work?
  - are there any alternatives I should know about?
  - if I wanted to do things slightly differently, could I still use the method/feature/option referenced in this solution? does it have any parameters that are easy to swap or tweak?
  - is this scenario what the feature/method/option in the solution is actually intended for? should I care?
... and the quickest way to answer questions like that is usually a glance at a manual.

-----

1: In some cases with GNU stuff the literal `man` pages are abridged versions of the `info` pages. But even then, the `man` pages direct you to `info` pages. It's not like they leave you having.

Re:We're Leaving Kubernetes by chipdart on Tuesday November 05, 2024 @06:45AM (#42050753)
> Uh, what? What man pages are you reading?

Every single man page out there leads to a user experience that is at best subpar.

> One of the reasons Stack Overflow is so popular is that people who can't/won't read docs can use it to have answers spoonfed to them (...)

Pause and look at what you're saying. Your only criticism of SO is how it improves the task of providing meaningful information to users.

The way you opt to spin improvements to user experience as "spoonfed" speaks volumes of your inability to understand the problem and the value you place on gratuitous ladder-pulling. You even contradict your remarks on man pages.

> Reading isn't a singular skill— each genre requires its own skills (...)

No. Writing is a skill. Producing content that the target audience is able to consume and brings value is a skill. The moment you, as a end-user, feel the need to hunker down and decipher arcane texts is the moment you should realize the documentation is bad.

Again, Stack overflow is widely used as ad-hoc crowd-sourced documentation for a reason. Some project maintainers even go as far as to make it their own channel to provide technical support. Why so? Do you honestly believe its because the whole world is not smart enough to read man pages?

Again, those who do not waste their time on man pages are the smart ones who put their own time to better use.

Re:We're Leaving Kubernetes by sgarland on Tuesday November 05, 2024 @09:07AM (#42051545)
> Your only criticism of SO is how it improves the task of providing meaningful information to users... the value you place on gratuitous ladder-pulling

How is wanting others to learn ladder-pulling? Also, how do you assume people will have this kind of information handed to them when the people who are interested in deeply learning stop doing so, die off, etc.? If you say AI, first of all, best of luck with the hallucinations, but secondly, who is going to work on and train the AI?

> No. Writing is a skill. Producing content that the target audience is able to consume and brings value is a skill. The moment you, as an end-user, feel the need to hunker down and decipher arcane texts is the moment you should realize the documentation is bad.

I think I see the root disagreement here. You continue to mention "value," as though reading is itself not valuable. Sitting down to read a work of fiction arguably brings no value to anyone (except perhaps the author and publisher), yet millions do it anyway. Similarly, if I find a way to do something, I usually want to know if there are also other ways, and if so, if they're better. There's not much "value" there most of the time, but it brings me happiness, and enhances my knowledge of the subject.

Re:We're Leaving Kubernetes by sgarland on Tuesday November 05, 2024 @09:01AM (#42051508)
> And indeed, a trip to StackOverflow never ends at StackOverflow for a person with much curiosity.

My favorite variety of SO question is "how do I do X in $LANGUAGE," because inevitably, people pile in with various answers, and then someone starts benchmarking all of them and providing graphs. Occasionally someone even breaks down the assembly instructions for each solution and explains why one is superior to the other. All in all, a fanatical obsession over something small and relatively unimportant, because they like to learn, and they like to share what they've learned.

Re:We're Leaving Kubernetes by sgarland on Tuesday November 05, 2024 @08:57AM (#42051477)
> Do you actually believe that you build up enough knowledge on [distributed services] to become a professional in the field if you "straight-up don't want to learn"?

Given the modern hiring practice of "can you pass Leetcode," and "can you memorize and regurgitate how to architect a link shortener," yes, yes I do. There is a vast difference between learning to pass a test, and learning because you're sincerely interested in the topic.

> Everyone wants to get paid. Do you know anyone who works non-profit?

Of course we all want to get paid. The intent of the sentence, as I think you know, was that many lack intrinsic motivation, of learning for the sake of learning.

> What you're failing to understand is the "do their job" part. Software developers are trained to solve the problems they face, and not waste time with the problems they do not have.

I think what you're failing to understand is that there is a difference between a factory worker and a craftsman. There is absolutely nothing wrong with factory work, to be clear here – I in no way intend to disparage honest work – I just personally find it a difficult personality to work alongside.

> Time is precious, and they invest it where it has the largest return on investment.

To me, this reads as "be selfish." The fastest way to get an answer is to ask someone who knows. This is not, however, the best way to retain knowledge, nor is it considerate of others' time. That's not to say you shouldn't ask for help, but it's a much different ask when you come to someone saying, "this is what I'm trying to do, this is what I've done, and this has been my result – can you help?"

I can't tell you the number of times someone has DM'd me asking for help on something I've never touched, but by reading docs, have solved. I always try to reinforce that by linking to the docs in the answer, but it hasn't proven to be a successful method of deterring future LMGTFY.

> Man pages are notoriously a colossal waste of time.

Citation needed.

> In general they are poorly thought out

Do you have some specific examples?

> They are incomplete

See above; also, if you've found this to be true, have you considered giving back by updating them?

> They were written with complete disregard for user experience

They were and are written for people who wish to understand their tools, not for people who want a 5 minute Medium post that contains the code necessary to complete a task.

> And more often than not they are way out of date.

I can't think of a time where the man pages _included with a tool_ were out of date. If your system is itself out of date, I can see where this could be true. Again, do you have some specific examples?

> Why do you think sites like Stack overflow is so popular? Because all those "incurious" people in tech feels the need to ask questions and dig through answers on how to solve problems?

SO is a great site, with a dizzying variety of quality in its questions and answers. Take one of (the?) most upvoted answers ever, on branch prediction [0]. The question itself isn't easily answerable via reading docs, and as the answer shows, is surprisingly deep. Next, a highly-upvoted question about how to reset local git commits [1]. This is a question that _is_ easily answerable by reading docs [2]. Or a question on what `__main__` is [3] in Python. A fair question (it is somewhat odd from the outside, especially if you have no experience in Python, have no idea what dunder methods are, etc.), but again, one that's easily answerable by reading docs [4].

> I think you're just picking a very personal definition of competence which conveniently boils down to "do the things I do, and do not do the things I don't".

Of course I think that the way I do things is mostly correct; otherwise why would I be doing them?

> Except the bulk of the people in the field is smart, and some have already solved problems that you aren't aware exist, such as wasting precious time deciphering unreadable documents that are systematically out of date.

Strawman aside, I never said people in tech aren't smart, I said they're largely incurious. Words matter.

[0]: https://stackoverflow.com/a/11227902/4221094

[1]: https://stackoverflow.com/questions/927358/how-do-i-undo-the...

[2]: https://git-scm.com/docs/git-reset#Documentation/git-reset.t...

[3]: https://stackoverflow.com/questions/419163/what-does-if-name...

[4]: https://docs.python.org/3/library/__main__.html

Re:We're Leaving Kubernetes by chipdart on Wednesday November 06, 2024 @02:28AM (#42057840)
> Given the modern hiring practice of "can you pass Leetcode," and "can you memorize and regurgitate how to architect a link shortener," yes, yes I do.

You are contradicting yourself. If there's anything that requires studying and preparation, that's leetcode.

Also, "memorize and regurgitate how to architect a link shortener" is also known as learning and knowing the basics of systems architecture and software architecture. That's an odd way of criticising others for being more competent than you.

Re:We're Leaving Kubernetes by saghm on Wednesday November 06, 2024 @04:04AM (#42058324)
> Of course I think that the way I do things is mostly correct; otherwise why would I be doing them?

"My way is correct" doesn't necessarily imply "other ways are incorrect". Sometimes there's not one single solution to a problem. I love using Linux for my personal machines, but I don't think that people who don't are doing things wrong; they just have different preferences on how to do things, and that's fine.

Re:We're Leaving Kubernetes by jrockway on Monday November 04, 2024 @08:06PM (#42047716)
I think this, or something of equal complexity, is probably the right choice. I have spent a lot of time helping people with their dev environments, and the same problems keep coming up; "no, you need this version of kubectl", "no, you need this version of jq", "no, the Makefile expects THIS version of The Silver Searcher". A mass of shell scripts and random utilities was a consistent drag on the entire team and everyone that interacted with the team.

I ended up going with Bazel, not because of this particular problem alone (though it was part of it; people we hired spent WEEKS trying to get a happy edit/test/debug cycle going), but because proper dependency-based test caching was sorely needed. Using Bazel and Buildbuddy brought CI down from about 17 minutes per run to 3-4 minutes for a typical change, which meant that even if people didn't want to get a local setup going, they could at least be slightly productive. I also made sure that every dependency / tool useful for developing the product was versioned in the repository, so if something needs `psql` you can `bazel run //tools/postgres/psql` and have it just work. (Hate that Postgres can't be statically linked, though.)

It was a lot of work for me, and people do gripe about some things ("I liked `go test ./...`, I can't adjust to `bazel test ...`"), but all in all, it does work well. I would do it again. Day 1 at the company; git clone our thing, install bazelisk, and your environment setup is done. All the tests pass. You can run the app locally with a simple `bazel run`. I'm pretty happy with the outcome.

Nix is something I looked into for our container images, but they just end up being too big. I never figured out why; I think a lot of things are dynamically linked and they include their own /usr/lib tree with the entire transitive dependency chain for that particular app, even if other things you have installed have some overlap with that dependency chain. I prefer the approach of statically linking everything and only including what you need. I compromised by basing things on Debian and rules_distroless, which at least lets you build a container image with the exact same sha256 on two different machines. (We previously just did "FROM scratch; COPY <statically linked binary> /app; ENTRYPOINT /app", but then started needing things like pg_dump in our image. If you can just have a single statically-linked binary be your entire app, great. Sometimes you can't, and then you need some sort of reasonable solution. Also everything ends up growing a dependency on ca-certificates...)

Re:We're Leaving Kubernetes by pxc on Tuesday November 05, 2024 @01:33AM (#42049091)
> I think a lot of things are dynamically linked and they include their own /usr/lib tree with the entire transitive dependency chain for that particular app, even if other things you have installed have some overlap with that dependency chain. I prefer the approach of statically linking everything and only including what you need.

Wherever there's such overlap, those dependencies are already shared. Static linking in such a situation means more disk usage, not less.

Packages in Nixpkgs have large closure sizes for entirely other reasons, like not splitting packages as aggressively as they could be split, or enabling/including most optional dependencies by default. Distros like Alpine typically lean the other way for their defaults.

It's true that if you're willing to manually mangle them, static binaries are nice because you can very easily strip all docs and examples or even executables that you don't need, and still know your executable will have the libs it's linked against. In one place at work I actually do this with Nixpkgs— there's a pkgsStatic that includes only statically compiled packages. I pull just the tiny parts of some package I need out and copy them onto a blank OCI image because it was the path of least resistance.

But Nix also has some really nice tools for inspecting dependency graphs to figure out why large packages are getting pulled in. nix-tree is my favorite, but there's also the older nix-du that gives the same info via graphviz instead of the terminal, and the built-in `nix why-depends`.

-----

Edited to add: wait, are you saying you used some other base distro to create Docker images where some things were supplied by Nix and others came from the base distro? If so, yeah, Nix is going to bring all the dependencies along, all the way down to libc or whatever. That's required for the kind of hermeticity that is its goal.

Mixed images like that are always going to be larger. But you also don't need a base distro at all with Nix. You can use one of the existing Nix libraries for Docker/OCI stuff to generate a complete image from scratch, or just copy your Nix packages' dependency closure onto an empty image with a FROM SCRATCH Dockerfile.

If you can't do that, you can do various things to try to slim things down but it's best to just Nixify whatever other packages you're using so you don't need a base distro. (And if you're trying to save space, Nix itself doesn't need to be in your Docker images either, which can also cut out some deps.)

Re:We're Leaving Kubernetes by jrockway on Tuesday November 05, 2024 @09:04AM (#42051525)
Just to answer your edit, nope, I wasn't mixing distros or anything. It would either be all Nix or all Debian.
Re:We're Leaving Kubernetes by pxc on Tuesday November 05, 2024 @02:48PM (#42054606)
Got it. Yeah, that would likely be easiest to resolve by exploring the closure with nix-tree and looking at how to eliminate or split dependencies. But pkgsStatic can be a handy fallback option too.
Re:We're Leaving Kubernetes by kstenerud on Monday November 04, 2024 @10:27PM (#42048331)
After my personal 2-year experiment with NixOS, I'd avoid anything Nix like the plague, and would be looking for a new job if anyone instituted a Nix-only policy.

It's not the learning new things that's a problem, but rather the fact that every little issue turns into a 2-day marathon that's eventually solved with a 1-line fix. And that's because the feedback loop and general UX is just awful - I really started to feel like I needed a sacrificial chicken.

Docker may be a dumpster fire, but at least it's generally easy to see what you did wrong and fix it.

Re:We're Leaving Kubernetes by otabdeveloper4 on Tuesday November 05, 2024 @12:59AM (#42048948)
> ...but at least it's generally easy to see what you did wrong and fix it.

You're not actually "fixing" anything, you're just passing the ball of shit down the responsibility chain to the ops/infra team.

Which is fine if you work in a large corporation where this is a valid strategy.

Unfortunately though the software supply chain problem is a) very difficult and b) unavoidable.

Nix is the best (or maybe only) attempt to solve this problem with programmatic (vs organizational) tooling.

Re:We're Leaving Kubernetes by kstenerud on Tuesday November 05, 2024 @03:54AM (#42049831)
I've got nothing against the fundamental concepts that Nix strives for. In fact, that's what triggered my 2 year journey with it. I just hate the implementation with a passion. The overall result is worse than before.
Re:We're Leaving Kubernetes by otabdeveloper4 on Tuesday November 05, 2024 @06:42AM (#42050734)
There is no possible way to solve anything in this problem space without triggering developer PTSD. The implementation is not at fault.

(See npm or the clusterfuck of Python packaging for proof.)

Re:We're Leaving Kubernetes by pmarreck on Tuesday November 05, 2024 @02:22AM (#42049340)
Literally every Docker image you've ever downloaded comes from a build that only worked (essentially) "by accident."

If Docker builds were as deterministic as Nix, then all that would need to be distributed would be Dockerfiles and perhaps a cache of base images somewhere.

Looking at a build as a pure function where each dependency (including any compiler(s), plus the environment), are "input arguments" to it, was a revelation (since I already realized the advantages of pure functions while working in functional languages).

Running a Dockerfile and hoping to get a working image out of it is like running a function which checks the time when it runs and errors when the seconds end in 0 due to a bug.

> every little issue turns into a 2-day marathon that's eventually solved with a 1-line fix

There is spotty education in the space. Did you ever take this (very cool) Nix tutorial? Not actually understanding Nix is going to make any troubleshooting of Nix much harder. https://nixcloud.io/tour/

> I really started to feel like I needed a sacrificial chicken.

Have you looked at Guix? A lot of people think it's "Nix without the warts." Plus it uses a Lisp, which some people prefer, or can at least grok better than the Nix language. https://guix.gnu.org/

Re:We're Leaving Kubernetes by kstenerud on Tuesday November 05, 2024 @03:59AM (#42049857)
I watched all the videos, read all the tutorials (including the "super easy" ones). I understand the Nix language, but the actual infrastructure is just so damn convoluted and fragile, the error feedback mechanisms are worse than useless, and the documentation assumes that you already know everything inside and out. It reminds me of the bad old days of the neckbeard gatekeepers.

I'll take a look at guix, though...

Re:We're Leaving Kubernetes by saghm on Wednesday November 06, 2024 @05:29AM (#42059484)
"Working by accident" is still "working" though, so from the standpoint of the person using it, the reason doesn't matter. I'd also argue that the main point made above about docker wasn't that it's reliable, but that it gives developers more feedback when trying to work with it. For people who's goal isn't just to consume builds but write their own, existing Nix builds being deterministic does nothing to help them if trying to modify or borrow parts of them leads to issues they can't easily fix.

I think Nix fits a pattern that's happened in plenty of other domains where the technology that focused on doing things "right" failed to win out against a competitor doing things "wrong" but optimizing for a lower barrier of entry. The logic that a perfect solution is worth an up-front cost is compelling, since having an imperfect solution has a long-term cost that never goes away, but this misses the fact that pushing the cost until later has value of its own; making things easier today at the cost of tomorrow buys time to improve things before the cost is incurred.

At the risk of a convoluted metaphor, imagine that someone moves into a new house and calls two plumbers asks them to hook up the water in their bathroom. The first plumber says that they can get it done so the bathroom can be used today, but they'll need to come by again in a week or two since they might need to make additional adjustments. The second plumber says they've come to with a way to make sure that they never need to come back to make adjustments, but it will take them a full week to finish setting it up before anyone can use it.

For most people, it doesn't matter if the second plumber's solution will be better next week if they need to use the bathroom today, as long as the first plumber's solution can last long enough before it needs to be fixed.

Re:We're Leaving Kubernetes by lemme_tell_ya on Monday November 04, 2024 @11:20PM (#42048582)
> time is possibly too short (especially if you have kids) to learn new things that aren't simple, even if better

Having a kid has drastically altered my ability to learn new things outside of work, simply due to lack of time. I never could have imagined how big of an impact having a kid would be, its crazy!

The worst thing is when you actually manage to carve out some time to do some learning or experimentation with a new tool, library, etc only to find out that it sucks or you just don't have the time to pick up or whatever.

Re:We're Leaving Kubernetes by pmarreck on Tuesday November 05, 2024 @01:44AM (#42049141)
> Having a kid has drastically altered my ability to learn new things outside of work, simply due to lack of time. I never could have imagined how big of an impact having a kid would be, its crazy!

yeah, I could have written this verbatim. Either I was not warned enough, or I did not pay enough attention/heed whatever I was warned of. I don't have a large family, so I've basically had ZERO kid experience since I was a kid... yikes... almost 50 years ago LOL. What worries me though is that it's kind of been an assumption at this job that you DO spend some off-duty time learning/tinkering. And I enjoyed it!

> The worst thing is when you actually manage to carve out some time to do some learning or experimentation with a new tool, library, etc only to find out that it sucks

I got briefly excited about the V language to maybe use for little utility scripts and maybe even as a first teaching language for my kid, then realized that when you scratch the surface of it it's basically kind of ugly underneath. (Example- The "this should never happen" error was literally most of the errors, lol.) It looks like something with a lot of great ideas but slipshod not-deeply-thought-out implementation. And the final nail in the coffin was all the evidence that the language creator simply bans anyone with valid criticism- I'm a free-speech near-absolutist so that one was the killer for me.

One of a few examples of what you're referring to. The thing is, before kids, we could afford to waste that time. Now we cannot. :/

Re:We're Leaving Kubernetes by otabdeveloper4 on Tuesday November 05, 2024 @12:54AM (#42048935)
> there is no (to my knowledge) general-purpose, broadly-accepted way to deploy via Nix

`nix copy .#my-crap --to ssh://remote`

What you do with it then on the remote depends on your environment. At the minimum do a `nix-store --add-root` to make a symlink to whatever you just copied.

(The most painless path is if you're deploying an entire NixOS system, but that requires converting the remote host to NixOS first.)

Re:We're Leaving Kubernetes by rtpg on Tuesday November 05, 2024 @06:04AM (#42050507)
I trialed for a job where the CTO was convinced of dev environments in kube as "the way to work". Everyone else was at least ambivalent. I joined, tried to make some changes that would let me run things locally. Every time I got pushback about using the dev environments instead.

It took me a couple of days to get a supervisor-based setup working locally. I was the only person on the team who would run the backend and frontend when trying things out, because nobody was actually using the dev environments fully anyways. There was no buy-in for the dev environment!

I really feel like if you are in a position to determine tooling, it's so much more helpful to lean into whatever people on the ground want to use. Obviously there are times when the people on the ground don't care, but if you're spending your sweat and tears to put the square peg into the square hole suddenly you're the person with superpowers, and not the person pushing their pet project.

And sometimes that's just "wrap my thing with your thing".

Re:We're Leaving Kubernetes by KronisLV on Tuesday November 05, 2024 @07:29AM (#42050998)
> I really feel like if you are in a position to determine tooling, it's so much more helpful to lean into whatever people on the ground want to use.

This might mean picking something that you think/know kind of sucks for the task, but that will be easier for most people to grok - while it might subjectively feel unfortunate, it's probably the right thing to do, for the sake of the majority of the team having an easier time.

Pushing your interests more strongly, or even in a top down fashion, might work, but that's more risky both in regards to letting everyone get things done, as well as team cohesion and turnover.

Re:We're Leaving Kubernetes by lmm on Tuesday November 05, 2024 @06:55AM (#42050813)
I love the idea/design of Nix and hope that one day someone will reimplement it in a way that one can reasonably understand/debug. Language is part of the problem but I think it's more of a https://www.lihaoyi.com/post/SowhatswrongwithSBT.html style problem where the execution model involves too much at runtime.
Re:We're Leaving Kubernetes by marcosdumay on Tuesday November 05, 2024 @11:26AM (#42052820)
> you either insist on it and say it's the only supported developer-environment-defining framework, or you will lose control over it

That's true for any architectural decision in an organization with more than 1 person.

It's really not something that should make you reconsider a decision. At the end of the day, an architecture that "people" actually want to use doesn't exist, "people" doesn't want any singular thing.