News for hackers. Stuff that matters.
Re:Why I self host my servers (and what I've recently learned) by kkfx on Wednesday September 04, 2024 @01:57AM (#41442351)
A small suggestion about resources: try using NixOS/Guix System instead of containers to deploy home services, you'll discover that in a fraction of resources you get much more, stability, documentation and easy replication included.

Containers now, like full-stack virtualization on x86 are and was advertisement stuff pushed because proprietary software vendors and cloud providers need them, other do not need them at all and devs who works for themselves and generic users should learn that: if you sell VPS et al. obviously you need them, if you made your own infra from bare metal adding them it's just wasting resources and add dependencies instead of simplify life.

Re:Why I self host my servers (and what I've recently learned) by snowpalmer on Wednesday September 04, 2024 @07:07PM (#41451800)
I agree that removing the container would be better on resources.

However, most self-hosted software is already "pre-packaged" in Docker containers. It's much easier to grab that "off-the-shelf" than have to build out something custom.

Re:Why I self host my servers (and what I've recently learned) by transpute on Wednesday September 04, 2024 @10:15PM (#41452864)
NixOS improves the reproducibility of both self-hosted software and configuration state.
Re:Why I self host my servers (and what I've recently learned) by kkfx on Thursday September 05, 2024 @03:45PM (#41459846)
In NixOS/Guix System there is no need of such package, the configuration language/package manager takes care of anything, configuration included.

Let's say you want Jellyfin?

    jellyfin = {
      enable = true;
      user="whatyouwant";
    }; # jellyfin
under services and you get it. You want a more complex thing, let's say Paperless?

    paperless = {
      enable = true;
      address = "0.0.0.0"; 
      port = 58080;
      mediaDir = "/var/lib/paperless/media";
      dataDir = "/var/lib/paperless/data";
      consumptionDir = "/var/lib/paperless/importdir";
      consumptionDirIsPublic = true;
      settings = {
        PAPERLESS_AUTO_LOGIN_USERNAME = "admin";
        PAPERLESS_OCR_LANGUAGE = "ita+eng+fra";
        PAPERLESS_OCR_SKIP_ARCHIVE_FILE = "with_text";
        PAPERLESS_OCR_USER_ARGS = {
          optimize = 1;
          pdfa_image_compression = "auto";
          continue_on_soft_render_error = true;
          invalidate_digital_signatures = true;
        }; # PAPERLESS_OCR_USER_ARGS
      }; # settings
    }; # services.paperless
Chromium with extensions etc?

    chromium = {
      enable = true;
      # see Chrome Web Store ext. URL
      extensions = [
        "cjpalhdlnbpafiamejdnhcphjbkeiagm" # ublock origin
        "pkehgijcmpdhfbdbbnkijodmdjhbjlgp" # privacy badger
        "edibdbjcniadpccecjdfdjjppcpchdlm" # I still don't care about cookies
        "ekhagklcjbdpajgpjgmbionohlpdbjgc" # Zotero Connector
        # ...
      ]; # extensions
     
      # see https://chromeenterprise.google/policies/
      extraOpts = {
        "BrowserSignin" = 0;
        "SyncDisabled" = true;
        "AllowSystemNotifications" = true;
        "ExtensionManifestV2Availability" = 3; # sino a 06/25
        "AutoplayAllowed" = false;
        "BackgroundModeEnabled" = false;
        "HideWebStorePromo" = false;
        "ClickToCallEnabled" = false;
        "BookmarkBarEnabled" = true;
        "SafeSitesFilterBehavior" = 0;
        "SpellcheckEnabled" = true;
        "SpellcheckLanguage" = [
                           "it"
                           "fr"
                           "en-US"
                         ];
      }; # extraOpts
    }; # chromium
Etc etc etc. You configure the entire deploy and get it generated, a custom live? With auto-partitioning and auto-install? Idem. A set of hosts in a network similar (NixOps/Disnix) and so on. The configuration language do all, fetching sources and build if a pre-built binary is not there, setting up a DB, setting up NGINX+let's encrypt SSL certs, there are derivation (package) per derivation options you can set, some you MUST set, defaults etc., it's MUCH easier than anything else, only issue is how many ready-made derivations are there, and in packaging terms Guix is very well placed, NixOS is more than Arch, even if something will be always not there or incomplete as long as devs do not learn alone the system and start using Nix/Guix also to develop, so deps are really tested in dedicated environments and so on, and users always get a clean system, can change and boot in a previous version and so on.
Re:Why I self host my servers (and what I've recently learned) by stackskipton on Thursday September 05, 2024 @09:15PM (#41461937)
Sigh Nix users.

I need to run uptime kuma, Here is Docker Compose: https://github.com/louislam/uptime-kuma/blob/1.23.X/docker/d...

What is equivalent in Nix?

Re:Why I self host my servers (and what I've recently learned) by zarzavat on Thursday September 05, 2024 @11:47PM (#41462710)
Nix/NixOS and Docker work in fundamentally different ways so you have to decide which method of operation is most suitable for your use case.

If you use Docker then each container requires its OS image, which is fine if you’re running on a server and running other people’s software, but if you want to develop your own containers on a laptop then that convenience becomes an inconvenience. Nix is more lightweight.

Re:Why I self host my servers (and what I've recently learned) by kkfx on Friday September 06, 2024 @01:04AM (#41463054)

    services.uptime-kuma = {
      enable = true;
      appriseSupport = true;
      settings = {
        ...
      } # settings
    };
These are examples, you have to write down the system configuration, meaning picking uptime-kuma, choose what deps you want and so on. No need to "run a oneliner" on a homeserver than ssh and do things by hand to tweaks, with no reproducibility or hoping someone else have already made them in something ready-made.

Doing the pick-a-oneliner is a classic developer in Silicon Valley mode move which cause a classic set of disasters, like wasting gazillion of resources for nothing, not seen an image with someone else ssh authorized key, keep running vulnerable software and so on because he/she just need a oneliner, no matter if it pull down 10 images, consuming 1Tb of storage, having stellar CPU requirements etc because you are in SV-mode, you do not care and anyway you do not pay for the iron because it's a company one or it's a cloud service you do not even see how much loads you generate, AWS bill will go to someone in accounting. On the other side of course IT business like that, because they can sell you a ready made image, selling just few line of code can't be priced much, while selling a service who provide a big file set might be priced well. Also selling a VPS and so on "hey devs, you literally just need to click here and we do the rest" pay very well. Those who sell of course. Aside such mindset create an enormous amount of layers who in the long run create so heavy and complex infra no one knows them.

Just try to see what Home-assistant do. By default their devs suggest to run it on a dedicated machine (!) so they imaging you buy a single-board computer juts to run a damn app, ending up with this model with a gazillion of single board computer scattering around. Second option containers. So to run HA you need let's say 1-3Gb of stuff, nearly no mention to the simplest pip install. Zero mention that unlike pip install NixOS does not need to download a gazillion of binary basic stuff for python deps, if you have them on your /nix/store you just link them on as many pkgs needs them, keeping just one copy.

In the end you craft your system, as a single application, a single configuration, which demand 1/10 of resources used otherwise, have much less thing who can breaks, it's easy to know entirely etc.

You might state that's a classic operation vs devs.

Re:Why I self host my servers (and what I've recently learned) by Arelius on Friday September 06, 2024 @02:17AM (#41463368)
I mean, I don't know what that is, but looking at the documentation, it should be something like:

  config = {
    services.uptime-kuma = {
      enable = true;
      settings = {
        PORT = 3001
      };
    };
  };
Re:Why I self host my servers (and what I've recently learned) by kkfx on Friday September 06, 2024 @11:04AM (#41466857)
It's a node.js web-app, so it might need some NGINX (or other web server) settings, eventual acme/let's encrypt etc, few SLoC to customize and that's the very point: being able to customize anything without two-stage with one by hand in a terminal or with separate tools like Ansible.
Re:Why I self host my servers (and what I've recently learned) by Arelius on Friday September 06, 2024 @01:47PM (#41468235)
Yeah, none of those seem to be covered in the parent post, so the Nix example is equally simplistic.
Re:Why I self host my servers (and what I've recently learned) by pxc on Friday September 06, 2024 @04:55PM (#41469653)
Support for uptime-kuma is built into NixOS and documented here: https://search.nixos.org/options?channel=24.05&from=0&size=5...

As to your question: on NixOS there's no need to fuck around with volume mounts or port forwards or specifying a source image or explicitly setting a restart policy, so the majority of what is defined in that file, which is pure boilerplate, disappears. And that docker-compose file doesn't configure any non-default settings or install any optional dependencies, so all you're left with for the NixOS equivalent is:

  services.uptime-kuma.enable = true;
so about 0.1x the amount of configuration as appears in that docker-compose.yml file.

But imagining for a moment that NixOS didn't already have built-in support for running uptime-kuma as a service, or even include uptime-kuma in its package repositories, I would still prefer to deploy it on NixOS, if I were deploying it for self-hosting purposes. Chasing the docker-compose.yml files of others is like distrohopping. The distrohopper bounces aimlessly from one operating system to another, his choice of fundamental operating system tools dictated to him by a string of out-of-the-box experiences which are determined by a complex enough confluence of factors that they are significantly random. When his distrohopping solves a problem, it causes another one, and his mode of engagement guarantees that neither problem is a thing he'll ever understand or have real control over.

Unlike the distrohopper, the distrochooser is patient. She is willing, when needed, to RTFM. For the price of learning how the software she runs is operated, and how the separate programs and libraries included in her OS fit together to form a stack, she is free. The distrochooser can run any Linux distro she wants, because she knows how to bring over whatever tools she needs. As a result, she has an opportunity that the distrohopper does not: the opportunity to choose basic tools— a package manager, an operating system, a deployment method, a configuration management system— based primarily on technical merits.

In so much of our lives, and perhaps especially our professional lives, we don't use the tech we use because its good. We don't use it because it's fun, or pleasant, or even reliable. We use it because it's bundled; the company already paid for it. Or we use it because of network effects: some application expects it, or it's the only platform some compliance-mandated tool supports, or its file format is widely abused for interchange because the vendor is a monopolist. Or we use it because it's the thing we already know and we're on a tight deadline.

Hobby computing is special. It's a context where there is actually room to choose our software for virtues that are intrinsic to it, or for reasons that are personal to us. Where we can choose tech just because it is beautiful, or because it was made by a friend, or because it feels good to use, or because it respects our freedom. To shrink our horizons when the code is transparent and freely available, when mastery is right there for the taking by anyone who wants it, is to erase one of the vital differences between the kind of computing that personal self-hosting is and corporate computing. It's a terrible waste.

I want to use NixOS because the iteration loop for working with it is fast, safe, predictable, and ultimately very satisfying. I want to use NixOS because it at least attempts to solve fundamental problems of building and distributing software that container-based solutions just punt on. I want to use NixOS because the interfaces it offers for managing services are outstandingly flexible and feature-complete. Why would I just surrender to worse-is-better, to network effects, to incidental bullshit— and give all of that up— when I can just roll up my sleeves and write the frickin' NixOS module? I can run whatever applications I like *and* run it on an OS I like, managed by tools I like. So why the hell wouldn't I?

Re:Why I self host my servers (and what I've recently learned) by pxc on Friday September 06, 2024 @03:37PM (#41469074)
> However, most self-hosted software is already "pre-packaged" in Docker containers. It's much easier to grab that "off-the-shelf" than have to build out something custom.

Imo, the quality and documentation level of an application's build system is a valuable signal in determining its overall health and competency. It usually (though not always) ends up being that well-maintained software written for modern runtimes is very easy to build from source and run. Even if I do end up using the developer's container image, I generally want to check out their manual deployment documentation.

Re:Why I self host my servers (and what I've recently learned) by otter-in-a-suit on Thursday September 05, 2024 @07:47PM (#41461436)
We use nix at work. I'm not a huge fan - I find it too opinionated. Appreciate it for what it is, though, and understand its fans.

Since at $work, we run K8s + containers in some shape or form (as well as in... basically all previous jobs), using the tech that I use in the "real world" at home is somewhat in line with my reasoning about why the time investment for self hosting is worth it as a learning exercise.

Re:Why I self host my servers (and what I've recently learned) by VTimofeenko on Thursday September 05, 2024 @08:36PM (#41461723)
Containers allow running software that does not have a nix package available and one could not be bothered to write. My lab is fully on NixOS, but a couple of services are happily chugging along as containers in podman.
Re:Why I self host my servers (and what I've recently learned) by kkfx on Saturday September 07, 2024 @04:57PM (#41476443)
So you prefer manually handling such containers, that you probably do not really know enough, perhaps even for serious usage, instead of writing a derivation? No reproducibility etc

For quick testing something, nothing to say, but for real, albeit personal homelab, usage...

Re:Why I self host my servers (and what I've recently learned) by pxc on Sunday September 08, 2024 @12:48AM (#41478268)
An escape hatch like that can be really nice for trying software out on a temporary basis, before you know if you care about it enough to write a package and a service module.

Given time limitations, I can imagine living with some applications like that for quite some time, their packaging sitting at the bottom of a long todo list. :)

Re:Why I self host my servers (and what I've recently learned) by kkfx on Sunday September 08, 2024 @03:10AM (#41478739)
Now dream, it's a dream, but perfectly feasible technically, where most devs have found and understood the NixOS/Guix System way to develop and manage whole deploy. We would have various distro with different configuration languages like Nix and Guix, and some less diverse, since all devs simply "package", meaning describing, their code in their own distro how hard could be port the derivation/description?

Try also compute how hw requirement for whole infra plunge thanks to an immense overhead avoided on scale. The sole unhappy? Those who sell services, because for that they do need containers. This would push another evolution: built-in easy to tune isolation from a config, so they can sell instead of a VPS a system where you upload just your config, they import it, producing an isolated FHS for you, something we already have (i.e. FHSUserEnv) but with tunable system isolation. You would get your VPS, configured with just few SLoC instead of manually/in wrapped ways via SSH, and they can sell much more on the same iron, with much better performances as well.

All of the above is perfectly possible TODAY, it's only a matter of mass knowledge or lack thereof.

Re:Why I self host my servers (and what I've recently learned) by VTimofeenko on Sunday September 08, 2024 @04:17PM (#41482861)
> manually handling such containers

Well not by hand of course. virtualisation.oci-containers is pretty darn good. Podman+systemd provides some local sandboxing. Network-wide firewall prevents odd traffic from happening.

> No reproducibility etc

Images can be pinned to specific versions providing some reproducibility guarantees. Same goes for configs mounted as volumes.

There is also some software (libedgetpu using bazel as a prime example IME) with a complicated build process. Packaging it is a major PITA and the nixpkgs issue[1] is a graveyard of attempts to do so. I just build it using a container and push the binary version to the node with coral tpu.

[1]: https://github.com/NixOS/nixpkgs/issues/188719

Re:Why I self host my servers (and what I've recently learned) by apitman on Thursday September 05, 2024 @09:52PM (#41462132)
I've heard good things about Nix's dependency management. Does it seamlessly handle the case where you have 3 different apps that all require 3 different version of python with different python dependencies?

Also, does it offer any isolation between apps and between apps and the OS, especially the filesystem?

Re:Why I self host my servers (and what I've recently learned) by Arelius on Friday September 06, 2024 @02:10AM (#41463343)
> Does it seamlessly handle the case where you have 3 different apps that all require 3 different version of python with different python dependencies?

Ohh yes! That is exactly what it does on a repeatable and documented way..

But it also does a lot of work to do this. And doesn't hide the systems of that so much. So that means you're likely to have yo care about how it does all that work. And if you start doing something weird, might have to do a lot of work yourself..

Re:Why I self host my servers (and what I've recently learned) by granra on Friday September 06, 2024 @02:47AM (#41463551)
The first, yes.

The second, you can do this by setting certain options in the systemd service for your apps, but this is true for any systemd distro.

Re:Why I self host my servers (and what I've recently learned) by kkfx on Friday September 06, 2024 @08:49AM (#41465680)
Yes by design, meaning you do not have an FHS structure where "installing" means copy this in /usr/share, this in /usr/lib ... Alla packages just see a small "virtual" root tree, meaning a network of symlinks.

You can choose for package A to se package B and C, but not D, different versions means simply different packages because you have package alone, let's say the current stable version but also package_major_minor for various supported major or minor version. As well as you can choose to fetch sources from a specific commit from a public repo, build them and link them in the system.

Every package is not "isolated" in the container self, like small userlands on a common kernel, but in the sense of "having different views of the common system", so you avoid wasting gazzilion of storage and ram and CPU plus anything is still from the upstream or yourself, there are no outdated forgotten deps left around.

Plus being anything configured in the config (well, not mandatory, but that's the typical way) you have a fully reproducible system, maybe not binary-reproducible meaning "packageA now is version x.y" when you rebuild the system, but anything could be rebuild in the current state of nixpkgs/guix states from few kb of text, typically versioned in a repo. Every update does not overwrite anything, if put some new versions in a special tree /nix/store o /gnu/store and updates symlinks accordingly, if it does not work you restore the previous state instead/before garbage collection.

Practically it do the same of IllumOS with IPS (the package manager) integrated with zfs (and the bootloader), here instead of zfs clones and snapshots you have a poor man's version with symlinks.

There is no isolation by default, meaning it's a single system, but you can "portion" the system in various ways. Of course on Linux there are no IllumOS zones of FreeBSD jails, the state of paravirtualisation it's very behind those unices.