News for hackers. Stuff that matters.
Re:Tarballs, the ultimate container image format by justinsaccount on Wednesday May 16, 2018 @12:54PM (#17083989)
Articles like this are pointless. I get that guix and nix are neat, and I think that every single time something about one of them is posted, but I don't have the slightest clue how to use either one of them.

Do you want to convince people that something like guix is better than docker? Then take something that is currently distributed using docker and actually show how the guix approach is simpler.

i.e. I have a random app I recently worked on where the dockerfile was something like

  FROM python:2.7
  WORKDIR /app
  ADD requirements.txt /app
  RUN pip install -r requirements.txt
  ADD . /app

  RUN groupadd -r notifier && useradd --no-log-init -r -g notifier notifier
  USER notifier
  EXPOSE 8080/tcp
  CMD ./notify.py
How do I actually take a random application like that and build a guix package of it?

Another project I work on is built on top of zeromq, and it would be great to use something like guix to define all the libsodium+zeromq+czmq+zyre dependancies and be able to spit out an 'ultimate container image' of all of that, but all this post shows me how to do is install an existing guile package.

Re:Tarballs, the ultimate container image format by t0nt0n on Wednesday May 16, 2018 @01:43PM (#17084561)
With Guix you get full introspection of your entire package dependency graph, you can check and manipulate every aspect - and it is still simple and easy to work with. With GuixSD you get this same introspection and overview, but of your entire system. creating a container, vm or even a docker image is a simple '$ guix system <container|vm> config.scm' away. And your config.scm is as complex as you like it to.

The simplest way would be to package the app for guix and you could just run '$ guix environment <name-of-package>' and you would be dropped into an environment with all your dependencies and whatever else the application requires in your path ready for hacking, get your sources and editor and start working.

If you need a vm or similar though I'd translate your example above into a system config where:

- packages include python-2.7 and whatever is in requirements.txt (this may mean you have to package a few things, but again this is usually super easy)

- users and groups are added to the config, as they always are, no extra step necessary.

- exposing ports and networking is available as options for qemu script guix produces to launch the vm.

- CMD ./notify.py: create a "simple" service that can be autostarted by the system on boot.

- filesystem access is also handled by arguments to the qemu script.

As always though there are several paths to Rome, and these are just two of them.

Zeromq and libsodium are already packaged on guix, czmq and zyre looks like they would be simple to package, guix is really quite simple to work with, which I think is the reason so many of the users and devs are running it as our daily drivers, even though it is strictly beta (0.14. I think is the last release).

And pointless, come on - what does that even mean? Does it mean you don't value them? I was quite happy to read about a neat new thing I can use my favorite tool for.

Re:Tarballs, the ultimate container image format by justinsaccount on Wednesday May 16, 2018 @02:05PM (#17084833)
> With Guix you get full introspection of your entire package dependency graph

Yes, I know all that. It's neat. I would like to learn more about it.

> The simplest way would be to package the app for guix

I was asking how to package the app for guix, and your response is the simplest way would be to package the app for guix...

> If you need a vm or soimilar though I'd translate your example above into a system config where: - packages include python-2.7 and whatever is in requirements.txt (this may mean you have to package a few things, but again this is usually super easy) - users and groups are added to the config, as they always are, no extra step necessary. - exposing ports and networking is available as options for qemu script guix produces to launch the vm. - CMD ./notify.py: create a "simple" service that can be autostarted by the system on boot. - filesystem access is also handled by arguments to the qemu script.

Yes, I'm sure it is super easy. How do I do it?

Do you know how to use the dockerfile I posted above? You run

  docker build -t myapp .
  docker run myapp
that's super easy. 9 lines and 2 commands. You can now add docker expert to your resume.

> Zeromq and libsodium are already packaged on guix, czmq and zyre looks like they would be simple to package,

Well, I was working on a fork of things, so I would have needed to install my forks.

> guix is really quite simple to work with

I'm sure it is!

> And pointless, come on - what does that even mean? Does it mean you don't value them? I was quite happy to read about a neat new thing I can use my favorite tool for.

You are correct, I don't really value posts saying how cool and easy something is and how much better it is than other solutions, when they don't actually present a complete solution someone can actually use.

I get that it is not other peoples job to teach me how to use something like guix, but do people not understand why things like Docker won?

Re:Tarballs, the ultimate container image format by t0nt0n on Wednesday May 16, 2018 @02:28PM (#17085091)
Right, your dockerfile contains a requirements.txt with unknown complexity and number of packages, your app is without a name and does not have any links to code.

I'd be happy to provide some examples. Say you want your fork of libsodium:

  (define-public my-libsodium
    (package
      (inherit libsodium) ; now anything not defined in this package will be inherited from libsodium
      (source (origin (method url-fetch)
                (uri "url-to-your-sources")
                (sha256 (base32 "hash"))))
     ; Add whatever other fields your fork needs.
  ))
Sure it's slightly more verbose. That's a bit of the cost of having something you can actually rely on, with that degree of hackability.

If you actually want help to package these things ask on our mailinglist or IRC, we're happy to help with specifics. But you're basicly complaining that I didn't give you a concrete solution to a problem with several missing details that are important. Docker would not be able to instantiate your python project if it did not know the contents of your requirements.txt.

The thing is docker is huge and bloated; is far from secure, and will probably stay that way for the foreseeable future; has a more or less complete lack of introspection; and is not strictly reproducible (sure, it gets quite far along the way, but it really is not).

Guix on the other hand is rather lightweight, and you have a fair amount of control over how lightweight it should be; builds from source, and has a sort of hotpatching system for security fixes; has introspection and is quite close to bitreproducible.

Sure, docker is _easy_, as long as it works. And I'd argue that because of its complexity and obscurity it is not practically free software.

Re:Tarballs, the ultimate container image format by cyphar on Wednesday May 16, 2018 @07:43PM (#17087367)
Regarding your concerns about Docker, I agree with that (even though I've been working on Docker and in the wider container community for almost 5 years now). However, there are plenty of tools that are compatible with Docker but provide similar benefits.

For instance, (from the openSUSE community which I'm a part of) we have KIWI that provides builds with full introspection on a package level (similar to what you're doing with Guix). If you build the image inside OBS (our build system) then if a dependency of your image is updated then your image will be rebuilt automatically and published in OBS (where it can be further pushed to any Docker/OCI registry you like). The packages are signed, and the image is also "signed" (though it currently signs the image artifact and doesn't use image signing since that is still not standardised). And most packages in openSUSE are bitreproducible (we build everything in OBS).

The above is far and above much better than the current standard in the "official" world of Docker, but unfortunately because OBS has a UI from the early 2000s (which is when it was written) it doesn't get enough attention outside of the communities that use it (and enjoy using it a lot). Everyone wants Dockerfiles even though they cannot provide these features (and you cannot get package manifests of your images without running a package manager in the image, which means you cannot get vulnerability information from the manifest).

[ Though I'm mostly talking about openSUSE here, I also happen to work for SUSE on the containers team. ]

Re:Tarballs, the ultimate container image format by pxc on Wednesday May 16, 2018 @09:46PM (#17088038)
> However, there are plenty of tools that are compatible with Docker but provide similar benefits.

And Guix is one of them, remember? From the article:

> Add -f docker [to your `guix pack` command] and, instead of a tarball, you get an image in the Docker format that you can pass to docker load on any machine where Docker is installed.

:-)

> The above is far and above much better than the current standard in the "official" world of Docker, but unfortunately because OBS has a UI from the early 2000s (which is when it was written) it doesn't get enough attention outside of the communities that use it (and enjoy using it a lot).

This is so true! I've mostly moved on from traditional, imperative package managers and associated distros in favor of the functional package management paradigm exemplified by Guix, but I still recommend openSUSE to my friends who prefer a more traditional/mainstream distro because of the love I have for the Open Build Service and Zypper.

The web interface for OBS does feel clunky these days, but it's a wonderful tool not just for improving the reliability and quality of software packages, but distributing them. Zypper is hands-down the most powerful and complete high-level package management tool I've ever used as part of a binary-based GNU+Linux distro. I love that openSUSE provides an instance of OBS that anyone can use for free to build packages for not just openSUSE but a TON of different distros.

I wish more people would explore, take advantage of, and celebrate OBS just like I wish they'd do the same with Nix and Guix!

Re:Tarballs, the ultimate container image format by justinsaccount on Wednesday May 16, 2018 @02:45PM (#17085257)
I don't think that's very verbose. The dockerfile I was using to build the app basically grabbed a specific version of all the deps and ./configure && make install'ed each one.

I'm completely onboard with the idea of reproducible builds.

> The thing is docker is huge and bloated; is far from secure, and will probably stay that way for the foreseeable future

It would be a mistake to fully associate container workflows with docker itself.

https://github.com/genuinetools/img + https://github.com/opencontainers/runc can be used to provide almost the same workflow as docker, without using docker itself.

You could also use the '-f docker' option the post talks about with runc to run the resulting image in an unprivileged container.

Re:Tarballs, the ultimate container image format by masklinn on Wednesday May 16, 2018 @03:52PM (#17085869)
> It would be a mistake to fully associate container workflows with docker itself.

It would also be a mistake to fully associate Docker containers with Docker.

Joyent can run docker containers, securely (zone-isolated), on smartos.

Re:Tarballs, the ultimate container image format by gfosco on Wednesday May 16, 2018 @02:05PM (#17084838)
I think you've reinforced the point they were making. It's pitched as easier, but clear examples of common usage aren't provided. You've provided a response longer than the 9 line Dockerfile, and we still don't know how to replicate it with guix.
Re:Tarballs, the ultimate container image format by t0nt0n on Wednesday May 16, 2018 @02:34PM (#17085149)
I thought giving concrete commandline invocations to be rather clear and precise.

I use 'guix environment <somepackage>' and 'guix system vm config.scm' every day. I don't need more, cause these two solves most of the problems that was described earlier.

What is it I can provide that would be clearer, more common usage, than the examples I use almost literally as they are here?

And that 9 line docker file references at least one other unknown file, and is part of a bigger program. Docker would not be able to reproduce with the information given in that post. How do you expect me to reproduce something with at least 2 huge unknowns?

That is why you got a more generic answer for implementation, but once you have your implementation once, you only need the commandlines I provided.

Re:Tarballs, the ultimate container image format by pas on Wednesday May 16, 2018 @04:38PM (#17086243)
That Dockerfile simply runs the commands listed therein in a glorified chroot, and then packages the result. The commands could easily be wget tar.ball && tar xf tar.ball && ./configure --prefix=/bla/bla/docker/ && make -j4 && make install

So, the question is, how to package something with guix, and how to run it.

With docker you run something as docker run [--interactive] [--terminal] [--entrypoint=...] <image> [[command] args]

Your libsodium fork example is nice, but we still don't know how to package a simple program.

Re:Tarballs, the ultimate container image format by tscs37 on Wednesday May 16, 2018 @02:38PM (#17085185)
To some extend I sympathize with GP because your post is exactly why I'm currently not using nix or guix.

While it's neat that I can do introspection on my package graph, I don't immediately see any benefit for me when I startup my containers.

I would love to see a full guix/nix script of what GP asked to see a comparison, I like to see hands-on stuff not theoretical.

Re:Tarballs, the ultimate container image format by rekado on Wednesday May 16, 2018 @10:34PM (#17088275)
> Do you want to convince people that something like guix is better than docker

No, we show that Guix is a tool that gives you a way to work with software environments at a higher level; but at the same time you don't have to give up on application bundles like Docker. You can simply generate Docker images or other forms of applications bundles from that higher-level representation.

You are welcome to take a look at this paper that I co-authored where we explain why we use Guix for a reproducible bioinformatics pipeline, and the rigorous, declarative functional package management approach instead of the imperative approach of Docker files:

    https://www.biorxiv.org/content/early/2018/04/21/298653
We're also providing Docker images, but we generate them from a higher-level declarative specification that ensures a high degree of bit-reproducibility.
Re:Tarballs, the ultimate container image format by Ruud-v-A on Wednesday May 16, 2018 @07:02PM (#17087160)
> it would be great to use something like guix to define all the libsodium+zeromq+czmq+zyre dependancies and be able to spit out an 'ultimate container image'

You define a package for your own project that depends on libsodium/zeromq/etc from GuixSD. Then you export your own package with 'guix pack'. For an example of what a package definition looks like, take a look in /gnu/packages in the GuixSD repository, for instance libsodium [1] or Vim [2].

I did something similar recently to build an Nginx "application bundle" [3]. It uses Nix (previously Guix, but Nix worked better for me in the end) to build a squashfs image. You can then run the binary on that filesystem with systemd-nspawn, or as a regular service by setting RootImage=. Some advantages over the Docker approach are that you can easily customise the build (e.g. changing the ./configure flags for Nginx without having to manually perform all other build steps), and bit by bit reproducibility (if you build the same commit six months from now, on a different machine, you will still get the same image out).

[1]: https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages... [2]: https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages... [3]: https://github.com/ruuda/miniserver#readme

Re:Tarballs, the ultimate container image format by t0nt0n on Wednesday May 16, 2018 @04:47PM (#17086324)
Packaged zyre and czmq. I'll send the patch for their inclusion, but until then here is the code: https://notabug.org/thomassgn/guixsd-configuration/src/maste...
Re:Tarballs, the ultimate container image format by myWindoonn on Thursday May 17, 2018 @10:57AM (#17091821)
From memory, not tested, not spell-checked:

    FROM nixos/nix
    RUN nix-channel --update
    RUN nix-env -i python2.7-{twisted,treq,txgithub}
    WORKDIR /app
    ADD . /app
    EXPOSE 8080/tcp
    CMD python notify.py
The next level would be using the nixpkgs Docker builder directly: https://nixos.org/nixpkgs/manual/#sec-pkgs-dockerTools
Re:Tarballs, the ultimate container image format by davexunit on Wednesday May 16, 2018 @03:53PM (#17085872)
It's hard to give you any specific recommendations with so little context, but I will try. For starters, I should point out that you can't really compare Guix directly to Docker. Guix is a package manager, Docker isn't. The article talks about 'guix pack', which makes it possible for Guix to interoperate with non-Guix systems, and one supported system is Docker. You can deploy software with just Guix, too, either on GuixSD or a foreign distro with Guix installed.

Anyway, in your Dockerfile I see that your application uses Python and you do some package management and service management stuff that is mixed together. In Guix, these things are separated. So the first step would be to define a package for your software, and then you would deploy that package. For a real world example of a Python application, here is what the AWS CLI package looks like:

    (define-public awscli
      (package
       (name "awscli")
       (version "1.14.41")
       (source
        (origin
         (method url-fetch)
         (uri (pypi-uri name version))
         (sha256
          (base32
           "0sispclx263lybbk19zp1n9yhg8xxx4jddypzgi24vpjaqnsbwlc"))))
       (build-system python-build-system)
       (propagated-inputs
        `(("python-colorama" ,python-colorama)
          ("python-botocore" ,python-botocore)
          ("python-s3transfer" ,python-s3transfer)
          ("python-docutils" ,python-docutils)
          ("python-pyyaml" ,python-pyyaml)
          ("python-rsa" ,python-rsa)))
       (arguments
        '(#:tests? #f))
       (home-page "https://aws.amazon.com/cli/")
       (synopsis "Command line client for AWS")
       (description "AWS CLI provides a unified command line interface to the
    Amazon Web Services (AWS) API.")
       (license license:asl2.0)))
The package recipe contains all the metadata, build instructions, and dependencies. Now that you have a package, it can be built with Guix and then deployed in a variety of ways. Judging from the Dockerfile, your software is some daemon that listens on port 8080, so:

* You can install the software directly using 'guix package -i your-package-name' and run the notify.py program. Good for trying things out.

* If you are deploying to the Guix system distribution, you could write a service definition so that you can manage the daemon via the init system. The service would take care of creating the notifier user and group, starting the service on boot, etc.

* You could use 'guix pack --format=docker' to export an image suitable for running with 'docker load'

* You could use a different 'guix pack' format (and maybe make it relocatable) for running on some other non-Guix system

I should also add that I don't think the work is fully done yet on handling the entirety of Docker use-cases. It's a work in progress. I can think of a number of things that I want to add to Guix to make this workflow better that I haven't had a chance to hack on yet.

Re:Tarballs, the ultimate container image format by justinsaccount on Wednesday May 16, 2018 @04:26PM (#17086146)
That's interesting, but where does it specify which python version is used and the version of all the dependencies?

If the versions are specified in the 'python-botocore' type definitions, how do you install more than one version of a library?

Does guix only track the latest version of dependencies or can you request any version of something?

Re:Tarballs, the ultimate container image format by rekado on Wednesday May 16, 2018 @10:28PM (#17088240)
Packages in Guix are just Scheme variables.

The package here uses the `python-build-system`, which defaults to the latest version of Python, but you can override that by specifying `(arguments '(#:python ,my-python))`, where `my-python` is a variable bound to a package value of the Python variant that you want to use.

You can easily install more than one version of a package as long as you have a package definition for it. You can install different variants (not just different versions) into separate profiles.

Guix is a Scheme library providing lots of variables that are bound to package values. These package values may have links to other packages (that's done with quasiquotation). Together they form a big graph of packages with zero degrees of freedom. Every version of Guix provides a slightly different variant of this package graph. When installing any package you instantiate a subset of this particular graph. Updating or modifying Guix gives you a different graph.

In order to keep things manageable we try to keep the number of variants of any particular package in Guix to a minimum, but you can install older variants by using an older version of Guix; or you can add new variables that are bound to package variants or different versions and install those.

It's very convenient and conceptually simple.

Re:Tarballs, the ultimate container image format by jancsika on Wednesday May 16, 2018 @04:16PM (#17086068)
@justinsaccount: can you give t0nt0n a clue what the contents of requirements.txt are, plus anything else needed to create a complete port to guix?

Then it would be great to see t0nt0n or someone else who knows guix do the port so we can fully compare these two approaches.

Re:Tarballs, the ultimate container image format by justinsaccount on Wednesday May 16, 2018 @04:42PM (#17086287)
it's not really application specific, just stuff like

  requests==2.18.4
the actual packages generally aren't important.

The cases were that would become interesting are where they require some C library dependencies first, like libpq-dev. In those cases something like guix/nix would be nice because it could be used to pull in the specific external dependencies as well.