From a technological perspective, everything you say is true, and FreeBSD is better in so many regards. But, compatibility and community support is just a deal breaker.

The jails vs docker argument is a good example: while technically better, Docker (or rather, cgroups and image distribution) have been standardised and have tremendous community adoption. FreeBSD cannot tap into any of this at all. Yes, jails might be better, but because they lack widespread adoption, they’re far less useful.

It’s a sad state of affairs, I wish things went otherwise, but I feel Linux is sometimes a good example of the “worse is better” approach at work.

I think one of the biggest mistakes the FreeBSD team made(in terms of more widestream adoption) was abandoning the Linux syscall table layer. It's still there, but it's stuck on 2.6.x last time I looked. It's a lot to maintain, yes, but it would help secure a lot more users like me, who are highly technical, potential contributors, who like a lot of the features(jails, better ZFS integration, easy to set up dtrace support etc), but also like to play around with bleeding edge software that tends to support Linux/OS X first, and maybe BSDs by accident, if you're lucky.

I think keeping this layer in development, maybe having a similar setup to SmartOS, letting you set up "Linux jails" or something like that, would significantly help adoption in the desktop/workstation space. This would of course lead to recruiting more developers, and so on.

Drivers are trickier, but I actually never had driver issues the last time I used FreeBSD as a daily driver(for about 6 months a few years ago, until I ran back to Linux with my tail between my legs).

It would be nice if at some point many decades ago, the Unix world had agreed on a standardised kernel API for modular drivers, so that drivers could more easily be ported between kernels, but it's way too late for that at this point I guess.

And maybe not even technically feasible, but I'm not qualified to comment on that.

> It's a lot to maintain, yes, but it would help secure a lot more users like me,

What an understatement. The modern Linux syscalls are tightly coupled to the implementation of Linux. Special fds, io_uring, robust lists, etc etc …

There’s a reason why Microsoft said fuck it and run Linux in a VM for WSL2.

If you want Linux, just run Linux.

This is exactly the kind of gatekeeping that holds FreeBSD back. Users asking about missing features being snarkily put down with things like "go use Linux then, we're doing just fine without feature X". This kind of toxic attitude is what pushed me away from FreeBSD in the end. They're so childish about constructive criticism sometimes, it's downright embarassing considering these are supposedly adults.

Edit: It's come to my attention that this particular instance of gatekeeping came from the Linux camp, not FreeBSD. Though you can find it in the FreeBSD community too, to be sure.

It isn't gatekeeping to point out that maintaining Linux syscall compatibility is impractical, especially citing the fact that Microsoft tried and gave up after sinking tons of resources into it; and FreeBSD has also already tried and gave up as you pointed out yourself.

Thanks for not calling me an idiot, at least.

I'm not sure how relevant Microsoft is here given how much more different from Linux NT should be compared to FreeBSD. It's also kind of a different calculus for Microsoft in terms of it being worth the effort for what they wanted to do with it. Microsoft already has its own thriving ecosystem, the Linux support is just a bonus to attract people from a niche market. For FreeBSD the situation is more or less the exact opposite. The SmartOS people did it too, and even ported over KVM. But Joyent was acquired by Samsung and I'm not sure what happened to SmartOS after that.

MNX acquired[1] SmartOS (and Triton DataCenter) earlier this year from Samsung. We're continuing development and have continued with a bi-weekly release cycle.

[1]: https://www.mnxsolutions.com/triton-faq

Awesome! Joyent had some really neat tech, I was sad to see them gobbled by Samsung.

You didn't really explain anything. You waved your hands a bunch. You didn't address the fact that FreeBSD already accomplished this, then stopped maintaining it, you conveniently ignored the fact that Illumos had it even later than FreeBSD, mostly from scratch. Your tone was overly dismissive. Now you start calling me names. Grow up. I'm sorry I misunderstood your point of view, but you did sound exactly like certain FreeBSD community gatekeepers, and you're responding to a post about FreeBSD project decisions, so is it really so strange or "idiotic"?

>abandoning the Linux syscall table layer

Assuming you mean linux(4), http://man.freebsd.org/linux, it's absolutely not abandoned: https://wiki.freebsd.org/Linuxulator; it recently got a proper ARM64 support for example.

>"Linux jails" or something like that

https://wiki.freebsd.org/LinuxJails

Awesome! Looks like they've picked up speed with it, that's great to see. Not sure if the abandonment was originally a misperception on my part or whether they picked it up again, maybe pulling code in from SmartOS? Either way the wiki says compatibility is up to 4.4 on CURRENT and 3.2 on STABLE. Still a ways to go, but that's sounding like I might have to give FreeBSD a go again!

Yeah, thanks to Dmitry returning to work on it, and FreeBSD Foundation sponsoring me, and other developers stepping in to fix something that annoyed them, and Alex tracking down Steam situation and doing epic work on tracking FreeBSD bugs and Linux idiosyncrasies.

Note that the reported kernel version doesn’t mean much, and we could probably bump it to 5. What matters is individual syscalls, and a nice thing about the state of Linux is that because it’s so fragmented, almost every non-portable syscall has a portable fallback.

I mean, it does work… I use it to run stuff like istioctl and rke on my fbsd workstation no problems…

Is cuda support decent on BSD?

Drivers for niche stuff seem like they would be an issue as well.

NVIDIA is the only GPU vendor who makes device drivers for FreeBSD, not only for Linux, so their GPUs work better than the others on FreeBSD.

Unfortunately the FreeBSD NVIDIA driver does not include CUDA.

It might be possible to use CUDA in a Linux VM with GPU pass-through, but I have not tried this.

FreeBSD has good driver support for many server-oriented devices, e.g. for many network cards, especially for those from Intel, and for many SCSI controllers. The support for magnetic tapes is also better than in Linux.

There are various random devices that may have good support, e.g. recording from a Logitech video camera worked much better for me on FreeBSD than on Linux, but in general the chances of not finding a device driver for some piece of hardware are much greater than on Linux, especially when that hardware is new.

No idea, never needed it. At some point I decided I had no use for dedicated GPUs that justified dicking around with atrocious driver blobs. That might change in the future if I get more into ML though.

I get that, mostly I use them for dsp for software defined radios. The drivers for those are probably hard to compile as well. Depedencies on libusb often. Or dpdk for the nice ones.

I'm still holding out hope that Intel will put out a decent dGPU with a strong ML story within a few gens, with gloriously open source drivers.

Recent news coming out of Intel the last week or so haven't been encouraging though. But I will remain a dreamer.

AMD is probably a better bet with ROCm although it's still pretty spotty right now in regards to support.

Btw, dunno much about software defined radios and such but I'm curious. Isn't this the kind of niche FPGAs are perfect for? Or do they not justify their cost in your use case?

Purchase price yes, development costs less so. Maybe I'm just bad at verilog but it takes me a lot longer to do the same thing compared to cuda or c++. Also the devtools are a big pain, and nothing is performance portable across device brands. Or even device classes. But yes, for high rate simple-ish dsp, fpga is awesome for performance and performance per watt.

Nobody ever adopts it because nobody wants to support it, ergo nobody ever adopts it and nothing ever improves. Gotta love this little cycle of mediocrity the software world has going on.

This type of circular argument sometimes hit a nerve when it about a thing cares about, but lets remember that the literal sense of "mediocre" is "middle" or "average", so its kind of a tautology to say that in general, software is average.

It takes true nerds (or "geek" in the sense it had around 2000, when it was still a bit derogatory and not yet a fashionable identity marker) to go off the beaten path.

Guess I mispoke, if everyone adopted it, then that quality of software would become the new average. this software is just inferior, which in software usually means that its harmful.

Adj. 1. mediocre - moderate to inferior in quality; "they improved the quality from mediocre to above average"

> Docker (or rather, cgroups and image distribution) have been standardised

Cgroups, namespaces have been implemented once, in Linux, by Google for their own needs. Not standardized in any meaningful interpretation of that word, just like "web standards" aren't. With now a generation of developers raised by big media and fake grassroots and staged march of progress stories, I'm wondering if that generation will be able to tell the difference, or are invested too much already.

Yeah, you’re right of course; my point was more that somehow we ended up with a “standard” format to store and distribute images, have plethora of tools that are able to run it, but they’re all using the same kernel fundamentals (cgroups).

But all that would be a distraction from the point I was trying to make, so I simplified it a bit.

What you're meaning is that "we" have converged on tech dominated by big cloud vendors. A standard, OTOH, is something implemented by more than a single party. Unix (POSIX) used to be standardized in that way. Portability, sustainability, and avoiding dominance by a single party was always the point of Unix; as on OS, Unix wasn't particularly innovative but merely good enough and compact, even when it was new.

Then what are foundations like CNCF (https://www.cncf.io/) doing? I'm asking honestly, if there's any difference between CNCF and any other standards governing body.

Perhaps VHS vs Betamax is a suitable equivalent to FreeBSD vs Linux.

Your comment implies the old trop that Beta was much better quality than VHS, but Betamax's reputation for better quality was marketing spin and conflation with Betacam, Sony's pro version.

In reality, Betamax was nearly indistinguishable from VHS.

Obviously it’s been quite a long time, but as someone whose first VCR was beta, this rings true. If nothing else, I definitely don’t remember a noticeable quality difference when we later switched to VHS.

and because jails has such low adoption, there will be fewer identified issues. Unlike docker, so many people use it, they will find many issues.

> backward compatibility, this is where linux is really horrible, there was an article about compiling binary on freebsd 2 and running it on freebsd 10. Try this on linux, binaries are not compatible even on minor versions.

Linux userspace is considered unstable. It's not a cohesive system like the BSDs but lots of independent projects all doing their own thing.

The Linux kernel is capable of running binaries compiled in the 90s. Breaking changes in the kernel-userspace binary interface makes Linus Torvalds scream at people on the mailing list.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

I even say that instability of the Linux desktop was a considerable contributor to the decline of Linux powered workstations.

People were getting used to the old reliable Gnome2 and the Windows-esque KDE desktop.

But there's always some team of developers thinking they are Apple and completely throw years of progress out the window by breaking compatibility because of some usability "feature" their own selfish conception of usability thinks is a good decision. Some of then even go to the point of rejecting Merge requests to their supposed open source software because of that kind of Whim

But why does the world always go along with it? In an alternate universe, the community just said "yeah, no thanks, I'll keep using Gnome 2", which they continued maintaining under the name Elf or whatever.

They did! MATE is fairly popular, but as to why it happens I think it's about people.

There are only so many people willing (and able) to do the often difficult and unglamorous programming work that goes into maintaining something like Gnome.

If you're working on a major distro you have a certain level of trust in Gnome 3 because the same people delivered and supported Gnome 2. I'm sure when Gnome 3's direction became clear dozens or hundreds said "no way! I'm going to start my own fork which will never change!"

But how do you know which one to pick?

The sad part is that Gnome 2 had a whole ecosystem of third party software that integrated with it. Just a few of that got ported over to Mate (usually just s/GNOME/MATE/g) but it never gained the same traction.

Why did it need to be ported at all? Shouldn't it "just work"?

Mate had to change all “Gnome” strings to “Mate” so both could be packaged/installed in parallel and probably also for legal reasons.

This means all DBus interfaces need to use the new name, a nautilus plugin must now call caja functions instead and so on.

...huh, well that's frustrating! Could this be solved with a basic compatibility layer?

or decompile, touch strings, then recompile.

Mate gives me a warm feeling of being at home where all is totally understandable and nothing has changed since 10+ years ago when Gnome 2 was a king.

I think that's where the fracturing comes in. For whatever reason, distros will ship with the new thing. The past 2 or 3 places I've worked adopted MATE (a continuation of GNOME 2). And, of course, there was instability at these places in figuring out which direction to go.

> For whatever reason, distros will ship with the new thing.

Chicken and egg problem: Popular distros will follow popular DE's, because they have the most people making their way out of bugs and problems.

MATE, on the other hand, is kinda barren compared to more popular options. Even if it's the same old Gnome2 desktop of yore

> The past 2 or 3 places I've worked adopted MATE (a continuation of GNOME 2).

Oh, so it did happen, I’m not currently a Linux user and had no idea!

For many years I was using gnome shell. Every update meant broken extentions. The system would also crash occasionally after few days on a multi-head setup. Then I switched to a hybrid of Sway and KDE and have been never so happy.

Yeah. Used to be a very happy KDE user too. Now I use i3, it's changed the way I use computers forever.

I wouldn't say "instability" as much as lack of focus

Especially as with some evolution it seems it's always a mix of people complaining new change means some X program made in the 70s will break and people who wants to redo something made in the last 6mo because it is "old" already and meet the newfangled thing which is 50% breaking changes 50% barely new things

I would say instability. Linux kernel will straight up revert improvements if it turns out they broke user space. Meanwhile in user space even glibc has broken binary compatibility before, GNOME breaks compatibility with user extensions...

> It runs circles around the docker in everything except how much it was adapted by community.

...except?

Meh. People who want to engage in this argument are generally trying to argue about jails vs. the collection of linux container technologies. That's not Docker. Docker won because of Dockerfiles. Docker isn't, at its core, an interesting container technology. Docker is a simple metaphor and programming environment to leverage container technology to solve practical[1] problems.

And as it happened, it was done on Linux and not jails, owing in large part to the more configurable/toolkit-style/policy-free tools available there. Jails were indeed more mature, but they were solving the wrong problem.

[1] Also thorny, boring problems like configuration management of large apps developed piecewise from components and by large teams. The kind of thing that is historically not well served by the operating system, BSD included.

I love FreeBSD and i love Jails, but you're right. Docker won because of simplicity.

Dockerfiles make it super simple. There is similar stuff for jails but you need a jails frontend that use it.

A few years ago i wanted to build the Jailfile equivalent, call it Warden, and have Warden manage the jails.

But honestly i doubt it would get traction outside a small community.

> Dockerfiles make it super simple. There is similar stuff for jails but you need a jails frontend that use it.

But docker is just a frontend for linux containers :) You're not comparing the right things here: Linux containers with a popular frontend and all the trimmings, to barebones FreeBSD jails.

Very true. I did mention that there are similar things with jail frontends, specifically iocage.

"Simple" is subjective.

Have you ever looked at what Docker puts in your iptables? I challenge you to understand what's happening with your networking after that.

If you want containers, lxc command line and a bit of configuration for a bridge interface is all you really need.

> If you want containers, [...] is all you really need.

And that is the attitude embraced by jails (to be clear: rather more cleanly and attractively than the expression of the same ideas in linux), and precisely why it lost.

No one wants "containers". They want docker. They want to be handed something that looks no more threatening than a mid-80's build script and have it magically be it's own little world with all its own software and versions and stuff, but still talk to the rest of the world on the same networks from the same piece of hardware (yes, via some weird voodoo to glue all that together).

It's like arguing that no one needs word processing because nroff or LaTeX is all you need. It's not wrong. It's just a failure in the market.

The voodoo is what bothers me. As the person responsible for the server, when things go wrong I will get blamed. In that situation, I at least want it to be my fault.

What "voodoo"?

Stock docker puts entries in your iptables which usually fully bypasses your firewall. try installing docker on stock Ubuntu, run a container with an exposed port, put ifw up blocking all traffic, and the container is still going to be reachable.

That's not voodoo, that's perfectly reasonable behaviour.

Why are you wasting time with ipfw?

> Have you ever looked at what Docker puts in your iptables?

Some forwarding rules.

> I challenge you to understand what's happening with your networking after that.

They're incredibly simple forwarding rules, to make sure that only the containers that you specify can connect to the Internet, and all the others in a particular group can only connect to each other.

> If you want containers, lxc command line and a bit of configuration for a bridge interface is all you really need.

Yes, assuming you never need to maintain it, or run more than one project per server.

CBSDfiles, reggae and bastillebsd can all build jails from a file based method

Build it and they will come..

More detailed comparison of Docker Vs Jails :

https://www.diva-portal.org/smash/get/diva2:1453017/FULLTEXT...

Remember, you're comparing Linux and FreeBSD in 2022 but BSD lost to Linux much earlier, many years ago. Back when I was looking into them (long time ago, excuse me for not remembering the details), BSD felt more pleasant and coherent. But at the same time it had limitations on scalability, performance and compatibility with hardware and also with userland software. In every benchmark, especially on multi-core, multi-socket systems, Linux was ahead.

My theory at the time was this: GNOME won on developers' desktops, so most software was developed on Linux natively, with BSD compatibility (and performance) as an afterthought. IIRC Linus made a similar point on the mailing list that developers love servers that resemble their programming environments. TDLR: BSDs got stuck in CLI-only mode for too long.

The more common explanation was that Linux got a head start by a few years by being a clean-sheet implementation, while the BSD had to spend its early years purging itself off the AT&T copyrighted code, so it was untouchable from a commercial use perspective.

I remember why I chose linux in 1998 for my desktop, and would choose for my server.

Hardware compatibility. I could install Linux on my shabby work desktop, and it just worked. Actually it worked more stably than NT 4.

Binary distros. I could apt-get install stuff onto my box in minutes. I rarely had to build things from source.

Speed of change. Linux was acquiring features at a breakneck speed. Large companies started contributing. SMP, interesting networking stuff, better disk I/O, new filesystems, stuff like that. Hell, Windows emulation good enough to run StarCraft! It felt alive and cared for. It was apparent that many serious businesses want to bet big on Linux. Some say marketing; I say GPL and project guidance.

I also had a lovely server box with FreeBSD. It had select compatible hardware. It had really nice documentation. It ran Apache and Squid pretty well. I had to build the latter from source IIRC. I had to build a lot from source (slow in 1998). If that was not available as a buildable package, I often had to tweak header files to make it build. For many amenities which I took for granted on my linux box, I decided that it's too much hassle to make them built on BSD.

Features like SMP or journaling file systems were a bit late in FreeBSD. Maybe they were more solid, and achieved performance parity with Linux with time. Sadly, the industry largely made the choice.

I also find modern Linux a mess, and run a minimalist distro (Void) on my laptop. I could consider running BSD on a server, but most servers now have to run VMs and containers within them, most tooling just assumes Linux.

> I remember why I chose linux in 1998 for my desktop, and would choose for my server. > > Features like SMP or journaling file systems were a bit late in FreeBSD

Yep, it was timing for me - the Asus BP6 allowed dual Celerons but only Windows 2000 and Linux 2.6 supported SMP. I was using FreeBSD at the time, but had to move because of hardware support.

But I wouldn't say it was hardware support alone - I think GNU/Linux won because of the licensing - FreeBSD not having a viral license would have turned people off contributing only to have their code distributed closed source by companies... that's my take anyway.

We're far from the days of getting a free Redhat CD in a book, just popping it in a drive and restarting to have it start a Linux install. You're lucky now if you don't have to enable USB booting, register a hash/key, turn on a 3rd party cert or just turn turn off secure boot to get a USB drive to boot.

I think it was too easy.

> The more common explanation was that Linux got a head start by a few years by being a clean-sheet implementation

My favorite theory for why Linux got a head start is in this (long) comment I found some time ago here: https://news.ycombinator.com/item?id=21420338

Some excerpts:

"With Linux, I just booted from a Linux boot floppy with my Linux install CD in the CD-ROM drive, and ran the installation. With BSD...it could not find the drive because I had an IDE CD-ROM and it only supported SCSI."

"It insisted on being given a disk upon which it could completely repartition. [...] Linux, on the other hand, was happy to come second after my existing DOS/Windows."

"By the time the BSD people realized they really should be supporting IDE CD-ROM and get along with prior DOS/Windows on the same disk, Linux was way ahead."

IDE was one of many hardware issues that just took too long to be solved. For a long time, BSDs didn't seriously try to support consumer-grade hardware - be it because of lack of manpower, conservative attitudes, or "commercial" choices. OpenBSD still doesn't support Bluetooth...

On the other side, the Linux community fought hard to get everything to work, creating positive loops: the more hardware it supported, the more people could get it to work on their hobby hardware, the more they'd become familiar with it and push for adoption at work.

>OpenBSD still doesn't support Bluetooth

OpenBSD supported Bluetooth but they threw out the whole stack because of security concerns. I think it's the right decision for OpenBSD.

Exactly, "commercial" choice. Any OS that wants to support consumer-grade hardware, in 2022, must provide Bluetooth; OpenBSD just doesn't care enough about the consumer market to do that.

True they don't care about you, it's like you don't count...who could imagine that!!!

Quel sacrilege!!

Lol, I think this comment illustrates why BSDs will forever lag, better than any technical explanation... and I say that as an OpenBSD user.

>BSDs will forever lag,

Terrible to hear that!

>and I say that as an OpenBSD user.

My condolence that you have to use that terrible anti-consumer system.

I have to get an outlet for my masochism tendencies somehow, at least this doesn't leave scars - physical ones, that is.

I have to say though, my insistent trolling of the project on forums like these occasionally contributes to the push to progress - like the long-requested syspatch. I'm not sure what yours achieves.

>my insistent trolling of the project on forums like these occasionally contributes to the push to progress

You really believe that right?

Almost as much as you believe this is a useful way to employ your time on the planet.

So you need Bluetooth and installed the only BSD who has no bt....sounds like a excellent use of time on this planet ;)

Sorry, I finished my troll-berries for today, see you on the next OpenBSD thread!

Sorry i use freebsd...the one with zfs and bluetooth ;)

Not sure what you could mean by “lost”, when BSD family operating systems are so widely adopted. I have more devices at home running such than I do Linux, not even counting Apple’s Mach hybrid. My internet traffic passes through more BSD-based than Linux-based devices as it crosses the globe, and my data resides on a variety of platforms that include both (and more besides).

These systems aren’t busy trumpeting their presence. Maybe the originators just don’t have fragile egos.

> My theory at the time was this: GNOME won on developers' desktops, so most software was developed on Linux natively, with BSD compatibility (and performance) as an afterthought.

when i tried FreeBSD a couple years ago i was actually surprised that gnome worked OOTB. i remember some of the settings were gated off (Bluetooth? can’t remember the specifics), but the base desktop worked exactly as before.

maybe different in the early days though: dunno, wasn’t there.

> jails... docker? Really? Jails are 15+ years old implementation, kernel supported, that stood test of time, actually being a security feature. It runs circles around the docker in everything except how much it was adapted by community. I never understood why people rather used an inferior solution like docker.

Docker has Dockerfiles, layers, and trivial push/pull of images. Compared to those workflow improvements, nobody cares if the guts suck.

> Compared to those workflow improvements, nobody cares if the guts suck.

Some people care, this is why FreeBSD is still around :)

And it's not really relevant if you build your own images anyway. And in fact within the scope of FreeBSD jails offer very similar features, one of the things that's very common to do is make a base image with ZFS and then base all your jails off it. This means you just have to update your base and all your jails are updated. It's similar to pulling the latest alpine with docker. If you embrace the full ecosystem with Bastille you will have very similar capabilities.

The big missing point is that you can't use images from docker hub. This is a big negative but if you already don't plan on using those it's not really a bad thing. And there is increasing resistance to pulling things made by unknown people into production (I'm sure many vulnerabilities will happen in the future as attackers start to take advantage of this).

But anyway it doesn't have to be for everyone. It doesn't have to be the biggest thing around.

I would like to be on the jails train, but the build/deployment story always felt inferior.

I have never seen anyone build jail-images from CI and deploying them to fleets of FreeBSD hosts. This may be technically feasible with zfs send/recieve but in practice people I know distributed packages (not images) to 10s of FreeBSD jails/hosts.

I want to deploy container images to a cluster - not install a package in a zone/jail. Is anyone doing this with FreeBSD/Solaris? (Outside of Joyent)

I want to deploy container images to a cluster:

Here you go:

https://bastillebsd.org/

#TIL Will check this out! Thanks!

- I can see the Dockerfile equivalent (Bastillefile). - Not sure about the image distribution (Docker repositories) and Cluster management parts (Mesos, K8S)

Or Pot:

https://pot.pizzamig.dev/

I've had a decent time writing little "jailfiles" and a create/destroy script. It's certainly nowhere near as ergonomic as docker though.

But it seems like a very solvable problem.

But, someone has to care enough to do so, and it has to catch on.

FreeBSD lacks the network effect.

Having used BSD many years ago on desktop, I completely disagree. What are you using your computer for using BSD? I really don't spend 99% of my time just compiling lol The chaos thing is something I hear from BSD users quite a lot. I'm still not sure what it is means, maybe that's some sort of personal issue..? I suppose you don't like forking and variation, which is understandable coming from a user of an OS with a fantastically small userbase that somehow still manages to be proportionally more fragmented than any other community I can think of, but the "chaos" of Linux is grossly overstated. Most everything that is relevant today runs Debian, unless you're a poweruser running Arch or even Gentoo, but even then, who's out there being a distro purist? I'm willing to bet that most people running Arch or Gentoo are still using quite a bit of 'Debian resources'/assets. Sure, there's RedHat too and all the others that fill some corporate niche, but there really isn't this whole divide within the Linux community like people sometimes imply. I think a lot of people are stuck in a mindset that hasn't been relevant for somewhere between 10 and 20 years.

I can’t speak for everyone, but one of the reasons I use Arch is precisely because Debian doesn’t have as many resources/assets as Arch.

AUR may not be perfect but there is very little software out there that someone hasn’t already packaged in it and it’s a great deal more convenient to use than faffing with custom repos.

The Arch Wiki is second to none too. I’ve even occasionally found myself using it when debugging other Linuxes.

I definitely wouldn’t use Arch on a server though. I mean, I have done, multiple times in fact, but they’ve always been personal systems. I much prefer FreeBSD for servers. Professionally it has usually been Debian, CentOS, or, much more recently, Ubuntu Server (but I really don’t like Ubuntu Server. Or Desktop for that matter).

Disclaimer: this is all my own opinion and I’m not trying to pass any of it off as fact.

Interesting. I suppose I also just use very mainstream software as well. I find that basically everything I need is either made for dpkg or exists within Ubuntu repositories (they've got literally everything there, haven't they!). I agree in your overall assessment though, I can concede to the fact that I might be underselling the value of Arch within the broader Linux community. I've often seen Arch as the detrimental opposite however, which is quite ironic. Personally I just find Debian distros superior to most others (except Linux Mint.I hate Mint, more than anything). I do quite fancy Ubuntu as dekstop, but you truly are right that it is at the very least, not a very good server distro. It might be the worst, imo, Debian server distro. My go-to choice for a Debian alternative is Fedora, however, so our different experiences might also be a testament to a fundamentally different view on Linux and how you do your computing.

> I'm willing to bet that most people running Arch or Gentoo are still using quite a bit of 'Debian resources'/assets.

More like the reverse, the Arch Wiki is the best so everyone uses it. Although for some topics it makes sense to take a look at Debian or Gentoo wikis, too.

Not sure what other kind of 'Debian resources'/assets you had in mind.

Whilst I don't at all doubt your testimonies, I am still sitting here wondering what programs everyone here is using, if not programs usually tailored for Debian. I feel like I've seen more software that runs natively on Debian (Ubuntu, really) and made specifically for dpkg. But I suppose a lot of applications also come with "snap" these days, but I avoid "software stores" like the plague. I personally have had very little contact with Arch, but I realise that is me that is strange, I personally dislike Arch very much and can recognise that, that might be the reason I don't see it around as much; a negative confirmation bias of sorts, if you will. 99% of the time when I have an issue and I just google it, I'll end up on the Ubuntu forum, that's no lie. But I do seem to remember using the Arch wiki years ago. I'll admit, I'm probably just a diehard fan of Debian.

You do know about package managers and software packages, right? Debian has its own packagers, and so does Arch, there's nothing in common between Debian and Arch packaging (except for certain elements of the philosophy, etc).

What do you mean by "programs usually tailored for Debian", in general Linux programs for sure won't be tailored for Debian, except for Debian-internal developer tools perhaps.

I use Arch almost exclusively on the desktop but I gave up on running it on servers that I don't want to actively maintain (think of small, non-critical web servers). Ubuntu works fine there.

Some of this may be true, though I have some disagreements about the accuracy of some (and also the issue that "Linux" is a broad target and some of these apply more or less to different distributions or os'), but freebsd definitely had its share if chaos in its history. The fbsd4-6 era was a difficult one to navigate and it's basically where I feel off the freebsd bandwagon.

Me too, at FreeBSD 6. ZFS was also pretty new and unstable in those days. I upgraded and could no longer read my pools.

But I came back to it in the last few years and it's matured a lot.

ZFS wasn’t just new when FreeBSD 6 was released, ZFS wasn’t even released itself. :P

FreeBSD 7 was the first release to support ZFS and even in that it was marked as experimental support.

So if you were using ZFS before FreeBSD 8, and especially somehow on 6 and below, then you’re very much using ZFS at your own risk and can’t really complain if things go sideways.

> backward compatibility, this is where linux is really horrible, there was an article about compiling binary on freebsd 2 and running it on freebsd 10. Try this on linux, binaries are not compatible even on minor versions.

What a completely bizarre claim. I have commercial linux/x86 binaries from the 90s that work perfectly well on my current PC running Linux 5.15.

I’m guessing the OP meant “where glibc is really horrible”, since bsd libc is rather more abi stable, largely because that’s where FreeBSD provides compatibility instead of the syscall layer. This is not to say you can’t use an old libc and fix up the loader paths or use a container and make it work, but it’s a different set of challenges.

> bsd libc is rather more abi stable

I am not sure about other BSDs, but at least OpenBSDs is all but abi-stable; when they wanted to introduce 64-bit time_t to fix Y2038, they just changed the typedef and declared a flag day.

I had trouble trying to run libc5 era software a few years ago. This wasn't the kernel but the distro did not make it easy. I wouldn't expect them to keep maintaining libc5 packages 20+ years later either.

I think if you do anything GUI-ish over the last 20 years there's been many shared library breaking changes. But that'd be true of FreeBSD too since those dependencies are just the same when they live in ports.

I use Linux all day long and none of this is an issue for me, so I guess that's the counterpoint. I'm just a desktop/server user/developer though and not a guru by any stretch.

Comparing Linux to FreeBSD is like comparing a Mercedes to a custom-made trackday car.

Docker, kubernetes et all was the nail in the coffin.

BSDs are amazing for universities, OS classes and some outliers in the industry here and there.

The one thing FreeBSD had was network stack performance. Linux supports higher throughput for a while now.. so even that ship sailed.

> Linux supports higher throughput for a while now

Really that's new...because netflix thinks otherwise.

> Really that's new...because netflix thinks otherwise.

That is true[^1]. Doesn't say much the two TCP stacks in 2022 though.

Netflix has built expertise around FreeBSD for years. Not just TCP stack optimisations. There's deployment, management, maintenance, etc. I don't expect them to jump ship easily, especially if the tool is good for the job - which obviously is.

However, even they reckon that FreeBSD _might_ be faster for _a specific_ workflow. Fifteen years ago there would be no comparison between the two in terms of top speed achieved on same hardware or latency, FreeBSD was far superior.

The trend is what's important here. Netflix choose FreeBSD in 2012, in 2022 it is certain that they would have gone with Linux and by 2032 my guess is that they'll be serving using Linux. I hope I'm wrong, the more the merrier.

[^1]: https://news.ycombinator.com/item?id=28584970

agreed ! Also i think throughput might be moving/dynamic values ?

i.e this year Lamborghini might be faster than a Ferrari... Next year they might swap. But no one will say Lamborghini or Ferrari are slow.

My humble point,unless absolute throughput (Then you looking at userland tcp-stack anyway) is your absolute goal. Things like tools,skillset, environment, community might play a bigger role. Yea i still love Freebsd but uses Linux on Desktop daily. Freebsd just kinda 'fits-in-your-head'

Linux: I miss the simple days when you could edit files like /etc/resolv.conf for nameservers :)

These days i see quite a few "don't edit this file manually it is auto-generated messages" :/

Thanks for the comment.

> ... /etc/resolv.conf ... ...

> ...don't edit this file manually it is auto-generated ...

I have been trying not to comment on any of this discussion, but You just hit a nerve for me (in a good way!).

resolv.conf: near and dear to my heart

Don't edit manually...: if it is cattle (most servers and IoT) - ok, I move on. If it is a pet (any of my main laptops), whoa. Stop what I'm doing and track down the software that did this. Purge it or re-write the offending code to not mess with stuff. Track down what was relying on this behaviour and purge. Rinse, repeat.

FWIW, I am running Slackware, OpenBSD, and sometimes FreeBSD on my pets (they are all installed it's just that I mostly live in the first two). Also, I find it hard to not have all the different man pages always available (no, I don't have a constant Internet connection - Security policies).

Your point about Jails vs Containers (popularized by Docker) is a naive rose-tinted glasses view. Jails is aimed at security and stops at that. To create something like docker-compose or Kubernetes with Jails is nigh impossible.

This "complete software stack" deliverables is what makes the difference today. You deliver software with greater ease. Jails isn't meant for that. Ignoring this point and claiming the same existed for 15 years is wrong.

> in next sub-version, the commands can have completely different switches

That claim seems crazy and doesn't match my experience at all. Please name a widely used program that changed its switches completely in a sub-version.

GNU ls adding quotes and you have to actually opt out

Do you mean displaying quotes in the output? That has been around for years (by default on many distributions) and doesn't answer my question at all. Please name a widely used program that changed its switches completely in a sub-version.

I too find this critique a bit odd.

Tho, switching between BSD and gnu versions of some tools, there is a mismatch of command line arguments.

IIRC I had to look up docs using tar and sed on BSD.

Honest question here about jails. I like docker because the tooling makes it super easy to get anything running in no time. How is jails tooling? Let’s say I need to get an app running, a database and a redis for the app. Is that as easy as docker compose?

I've run into fun issues like FreeBSDs slab allocator not playing nicely with the ZFS/NFS workload on one specific server and having to drop down to having ZFS allocate memory in a different way that used about twices as much CPU to avoid random multi-second pauses on the whole system. Also had issues where kerberized NFS would get confused and something would crash so had to have random cron jobs to restart dying components, since FreeBSDs init system is so simple. Also arbitrary and short name length limits, I think in both bhyve in the past (couldn't use fqdns for VM names since that would have exceeded the limit) and something about zfs mountpoints ages ago (I think we ended up changing our naming and nested conventions to work around)

I’ve noticed this myself but didn’t dig into the delays I’ve seen. Interesting to learn it appears for others.

How did you have ZFS change how it allocates memory?

This was a bug in ZFS itself, and was resolved years ago: https://smartos.org/bugview/OS-6404

In FreeBSD 12.x and older, you could opt out of using UMA with: vfs.zfs.zio.use_uma=0, but that is not necessary

Thanks. I still see multi-second pauses moving data over Samba to an encrypted ZFS pool. I’m using a 10Gbps network card, so the drives are the bottleneck (or so I thought, until seeing these delays).

I’m on 13-RELEASE.

> it works faster on my i5 server than kvm on my i7 laptop

Laptop processors are usually slower, due to lower thermal and power limits, so that's not surprising at all.

> it works faster on my i5 server than kvm on my i7 laptop.

What benchmark and what processors exactly?

I use Linux as my main OS. I’d love to switch to FreeBSD but I can’t.

Most software that run on Linux may or may not run on FreeBSD. And when it doesn’t, it’s typically hard or impossible to get instructions or community support.

Looking at the download page, Firefox doesn’t even seem to have a FreeBSD version or instructions.

I can understand for servers although you lose community support for most software and can’t use ready to deploy docker images, as much as I loathe docker.

FreeBSD is vastly superior to Linux IMHO (the handbook is a great read), but it’s unfortunately not practical for most usages.

>Firefox doesn’t even seem to have a FreeBSD version or instructions.

Erm, what? It's not Windows, that's what packages are for. Just "pkg install firefox".

I just checked the Firefox download page which doesn’t have any FreeBSD specific instructions (although it does for linux)

Glad to hear Firefox has up to date packages in FreeBSD though!

> I just checked the Firefox download page which doesn’t have any FreeBSD specific instructions (although it does for linux)

It's a Browser, you don't need "specific" instructions for the base-os.

You can do FIBs with Linux network namespaces, can't you? `setfib` would be `ip netns exec`

Aside: the thing that annoys me the most about FreeBSD is how bitter its proponents are that Linux "won". The tone of your comment a great example. It seriously turns me off. Some humility in finding out the shortcomings of BSD and ironing them out would be more beneficial in creating more competition in the OS space than complaining that all Linux users are idiots that haven't seen the light.

> I will never understand on what technical merits people are using linux for servers except the support-ability of hardware.

System upgrades are cleaner in Linux. Debian has had pkgbase ever since I can remember and it’s still underway in the BSDs. Live kernel patching is nice, too. Just two examples.

Docker is not a security sandbox technolgy, but a bundle all the dependency sandbox one.

With Linux today I can mostly use the terminal just for what is best done on the command line. With FreeBSD I always feel like I'm digging through the filesystem and terminal to get basic OS stuff done.

Don’t forget the docs are actual documentation. Like it is an actual coherent operating system with actual documentation. No “refer to info docs” here. Actual docs. Real ones.

As someone who has just been tearing my hair out over v1 and v2 cgroups and containers for testing systemd services, I think I need to spend some time in FreeBSD land.

bhyve (a lightweight virtualization tech) vs kvm (really kvm/qemu) isn't really a like for like fight. If your looking for something lightweight on the linux side you would go with Firecracker (AWS developed for Lambda and Fargate).

Not necessarily; if a VM is properly configured userspace is out of the way as much as possible and performance is dictated almost entirely by the processor and the hypervisor, not by QEMU vs. Firecracker.

Jails was actually committed to the source tree in 1999… so almost 25 years. You should read phk’s article - Jails: Confining the omnipotent root. It’s very good.

jails and lxc are great but Docker won out because they made those features significantly more user friendly and spent serious (VC-supported) coin on marketing that.