LXD is indeed nice, but it still isn't very widely adopted. I believe this comes down to 3 main issues:

1) It's strongly connected to Canonical and Ubuntu. This is mostly a matter of perception and it is an actual community project. However, I can understand people not feeling comfortable with "snap install lxd".

2) It sits somewhere in between k8s and docker engine. Over time, it will probably get more k8s-like features but still it is a weird position to be in.

3) It lacks a reach ecosystem of tools supporting it and a web UI. This makes it hard for newcomers to adopt. We're working on a web UI ourselves as part of our open source cloud management platform (https://github.com/mistio/mist-ce) and love to hear your thoughts.

Indeed, we have major issues with snap (doesn't work with $HOME on nfs, and auto updates with very few controls are a terrible idea on servers) so avoid anything dependent on it. Otherwise, I really like the concept.

It's unfortunate that the effort to get LXD packaged natively in Debian [1] seem to have stalled as this is definitely one of the last remaining drawbacks for me.

For example, when LXD was still available as a .deb from upstream, it was possible to run LXD inside a container and do nesting, but with the snap, that isn't possible anymore. (However it is now with the --vm flag, though that's really a different mechanism.)

As I understand it the big remaining difficulty is dqlite.

[1] https://wiki.debian.org/LXD

I did write a bit about my experience packaging LXD for Arch Linux a while back.

I got a ping from a Debian dev telling me the Debian dev working on LXD enjoyed the article, and has been trying to pick up on the work again. The linked wiki page is the outdated one.

https://linderud.dev/blog/packaging-lxd-for-arch-linux/

it's packaged for void, though I'm not sure what that entails. will say it works well enough I didn't find out about the snap mess until I tried to install it on another distro later

Debian has stricter rules regarding packaging and vendoring dependencies. Void and Arch packages LXD in a similar fashion by separating all the C dependencies. However, none of us actually separate out the different Go dependencies, they are all vendored inside the LXD package.

Debian separates out all of these into own packages.

I made a GUI too, but dont actively develop it anymore. https://github.com/dobin/lxd-webgui

As it requires docker probably difficult for us to try. Do you have any instructions on how to install it on bare metal? Using them can convert them to LXD container images to run, can reverse engineer Dockerfile to create LXD containers but try to avoid environment variables for configuring and running services as they are kind of security vulnerabilities.

Unfortunately we currently don't. We do have the option to install it on k8s with a helm chart though https://github.com/mistio/mist-ce/tree/master/chart/mist

I wonder how does Mist compare with OpenNebula, which seems to be in the same area? From a quick glance it looks more complex. (OpenNebula supports LXD directly.)

At a high level, Mist is a more "general purpose" platform which supports more providers (20) and more workflows. This increases the perceptual complexity, however it isn't more complicated to use than your average public cloud console. In fact, we strive to keep things as simple and as agnostic as possible. I'd be happy to arrange a demo if you like to see it in action. You can reach us on Github or from our website at https://mist.io.

OpenNebula definitely needs to revive the libcloud interface, or similar; I was thinking of complexity in operation more than using it. I ask mainly for general interest, as I can't think a free software management system would end up expensive enough for an institution which is too broke to retain staff. I might have a play sometime, and thanks for the response. I haven't found anyone with experience to compare these sort of systems in operation after seeing the need a while back.

> 1) It's strongly connected to Canonical and Ubuntu. This is mostly a matter of perception and it is an actual community project. However, I can understand people not feeling comfortable with "snap install lxd".

Last time I tried it on Fedora it did not work (less than 6 months ago).

Also it offers nothing I want over podman with --rootfs

I've tried both podman and lxd with success but I'm curious, what do you use a tool like that for, mostly?

Not to seem like a hypeman for Kubernetes and similar tools, but I actually seem to only ever use containers combined with something like Kubernetes or Docker Swarm. What do you do that you want to do specifically on one machine? Hosting something? Automation à la CI/CD?

Again, I am actually asking for good use cases without orchestration platforms, I am just curious.

I use it as a replacement for situations where I used to use KVM for Linux VMs. For example, my tvheadend and Zoneminder servers are both running inside LXD containers so that I don't pollute my host machine's environment. It's also a nice way to try out another distro other than what the host machine runs with close-to-metal performance.

One thing podman is used for is bootstrapping environments for package building under mock, which at least affects people doing Fedora maintenance.

> I've tried both podman and lxd with success but I'm curious, what do you use a tool like that for, mostly?

I use podman for dev and test environments, CI and CD workers and testing OCI containers that I eventually deploy in K8S. No production use cases. Hoping to soon see K3S working inside podman though, and then I would use it for deploying K3S :)

That isn’t a good comparison. While podman can run systemd inside a container, it isn’t widely adopted in the images in docker hub and elsewhere. There is probably just fedora supporting this. Whereas in LXD it’s normal to run a full systemd inside a container.

> There is probably just fedora supporting this.

How do you mean? I've used podman on Void Linux, openSUSE, GitLab CI, and I think some others that I'm forgetting (I distro hop a lot) and it's worked great.

> While podman can run systemd inside a container, it isn’t widely adopted in the images in docker hub and elsewhere.

With podman and rootfs it's also normal to run a full systemd inside a container and you don't need special considerations from OCI images for rootfs to work just fine.

> There is probably just fedora supporting this.

RHEL is behind podman and I will take RHEL support over Canonical every day.

Podman is also available on most major distros and easy to port to new ones without requiring someone to use some proprietary crapware solution like snappy.

> LXD is actually a cool technology.

It is a pretty nifty idea but like all things made by Canonical it is basically digitized garbage.

I tried to get LXC running on fedora some months, wondering why there are no official packages and then I very quickly noticed why. LXD uses the old Ubuntu trick of calling sysv style init scripts from systemd unit files without so much as a "|| exit 1" [1][2] - and I don't have time for that bullshit.

If you want vastly superior quality try podman with --rootfs or virt-install. At least those are not made by people with a disdain for error checking.

[1]: https://github.com/lxc/lxc/blob/master/config/init/systemd/l... [2]: https://github.com/lxc/lxc/blob/master/config/init/common/lx...

> It is a pretty nifty idea but like all things made by Canonical it is basically digitized garbage.

> At least those are not made by people with a disdain for error checking.

Every single time there's a topic about Ubuntu or Canonical you seem to go straight into offensive mode. I can tell you use Fedora, but that's not typical of Fedora users and developers and no excuse.

I'm an open source developer and believer, but this kind of behavior slowly burns my soul, even more when it seems accepted. I've worked on enough projects in my life that it's absolutely certain that you use my code regularly. It's inside Go, Python, APT, and RPM by the way (hello Jeff Johnson, wherever you are), and in key libraries that you surely depend on as well. And I never heard you complaining about any of that with such anger here.

I'm also Canonical's CTO, and I was one of the key designers and developers that started snaps, and juju, and other key projects from Canonical. I usually hear such blind hate in silence, but sometimes it's just too much. I don't understand why do we do that to ourselves, as a Linux community. Why is it okay to openly offend unknown people that we almost certainly depend upon? What is it that we came here to do, again?

You're confusing LXC with LXD.

LXC is legacy and is mostly left as-is for existing users who are happy [and don't want the LXD paradigm shift]. Think of it as 1.0. You're complaining about stuff that is effectively a maintenance branch. At least your links point to LXC, not LXD.

LXD is the new thing. Think of it as 2.0 and onwards.

I'm not sure if that is a good description. LXC is a library and set of command line utilities to run containers. LXD is a clustered service that makes use of the LXC library.

I admit it is confusing. LXC typically means the lxc-* commands like lxc-start and so on. That's the daemon-less "old way". The links the grandparent provided was to this, and their complaint is about that stuff that is in maintenance mode and not actively developed any more.

LXD is a complete rewrite. There is now a server daemon called "lxd", and the client for that is "lxc".

Also superior Proxmox

Proxmox is an absolute great product, in production at 7 clients since over 8 years, some with ZFS some with Ceph, some with lxd some kvm, it's rock-solid and the support is great!

Yup. I discovered LXC container with proxmox, now i have many of them running in my network.

Yeah, been playing with proxmox on my homelab, put it on an AMD Ryzen 5 3600 (6 cores, 12 threads), with 32gb of RAM and it's been running great with containers for when I want to fool around. Fantastic piece of software that has matured well.

Thanks! That's good to know because I've been having troubles with Ubuntu 20 and have been considering switching back to redhat.

Then the only remaining part would be figuring out how to do zfs on root in redhat. I do it with a chroot bash script for Ubuntu server atm, but hopefully there's a debootstrap kind of thing on redhat that I can use as a base.

> I do it with a chroot bash script for Ubuntu server atm

I’ve been using ZFS root on Ubuntu without any such nonsense for years, but I had to debootstrap it manually.

See guides here: https://openzfs.github.io/openzfs-docs/Getting%20Started/Ubu...

With Ubuntu 20.04 the standard installer should support setting up a ZFS root without all that manual work, but I haven’t tested it myself.

Yes, that's what I meant. The script does the partitioning, zfs init, chroot, debootstrap, and apt install. You just run it from a live cd.

https://github.com/kstenerud/ubuntu-server-zfs

> I do it with a chroot bash script for Ubuntu server atm, but hopefully there's a debootstrap kind of thing on redhat that I can use as a base.

There are ways to do chroot installs on RHEL but sadly not well documented.

Some resources on this:

- https://glacion.com/2019/06/16/Fedora.html

- https://github.com/glacion/fedora-chroot-installation

- https://web.archive.org/web/20150912004438/https://wiki.cent...

Most of that will work for RHEL with some minor modifications.

Pity it is not better documented.

YUM/DNF has bootstrap-in-chroot functionality built-in.

Just set up your disk layout the way you want on a disk device or a loopback disk backed by an image, and do something like:

$ sudo dnf --installroot=/path/to/mounted/target group install @core @hardware-support

And you're good to go. You can see the list of groups available to include by using "dnf group list --ids".

> And you're good to go.

If you opened the first link I shared you would see more is needed than this, which is why I linked to something explaining both this and the additional steps needed.

In fairness, distros seems to make this kind of "joke" one another.

About six months ago I tried running SystemTap on Ubuntu...

Calling sysv style init scripts with no error checking from systemd unit files is a joke on Canonical, not any other distro.