I had the same experience around the same time (though I'm a bit older). LFS really helped me understand the relationship between the bootloader, kernel, init system, and all of the various daemons and tools that make up a complete system.

From there I moved to Gentoo, and then eventually to Ubuntu and other batteries-included distros. But the knowledge I gained from LFS is still a foundational part of my skillset, even though my roles have been more dev-focused than ops/infra.

Until I moved to FreeBSD in the mid 2000s I ran Gentoo primarily. Even though it had a bit more hand holding than LFS, it still taught me a lot and getting to choose implementations allowed consideration of design choices (cron, mail, syslog, etc.).

I’ve tried to return a few times but it’s a bit different now and I’m happy enough with FreeBSD that I don’t have much place for it. I do wish more embedded distros would have started with Gentoo (specifically ARM SBCs).

I think one of the advantages of Gentoo or LFS is you get an “introduction” to a bunch of the various parts, so that when you need to change something in the future you know where to start looking.

Somehow LFS didn't help me understand the various subsystems... all I remember is long sed patches and compilation logs

It did teach me how subtle a running stable OS is underneath though, as, after my first standalone boot, I could enjoy a partially working TCP stack: elinks could browser, curl couldn't, irssi sometimes, all of this with random terminal display codes being inserted.

IMO a secure server is an immutable appliance. I still build Linux from scratch most weeks as a core part of my job.

Bare bones hardened kernel + shim init + target application are all you need, and will be your highest security/reliability systems.

And yeah, threadrippers are a must.

I assume you compile the source code because you want to be sure you don't use any compromised binaries? But how can you be sure the source code wasn't compromised with some obfuscated C code? (Honest question, I'm just a humble application developer.)

While I do not build LFS regularly or for production use, the security improvement typically comes from the fact that the end system is _super_small_ and focused. Less software means less attack surface.

Sure, compromised binaries are nasty but personally I do place quite a lot of trust with the distribution repos.

(PS, if you are reading this and contribute packages to distribution repos: Thank you!)

You can never be 100% sure. Even the compiler, firmware, or hardware could be compromised.

Security comes down to reducing attack surface, ideally to an infinitesimal degree.

It is dramatically easier to hide malware in a compiled artifact than in public source code, not to imply that the latter does not happen.

In security focused orgs though you review all code yourself with the exception of things with extensive third party signed review such as the Linux kernel itself. Even then I review codepaths in the kernel critical for my use case such as random.c

From there, if I -alone- compile containers, kernels, or binaries, someone could coerce me to tamper with them to compromise all downstream users. Same if there was a central build system I can access. To mitigate this I ensure my artifact builds are deterministic, sign my changes, and have team members review my changes, reproduce my artifacts bit for bit, then counter-sign the results.

It is never wise to be in a position where there is possibility of you yourself tampering with things that control anything of value, or else someone will coerce you to help them steal said value.

As a security engineer it is my job to ensure no one ever has to trust anyone, including me.

Typo in your homepage: "Continuious Integration"

Interesting thread!

Good catch!

As in manually or https://buildroot.org/ ?

I used to make heavy use of buildroot but these days for most of my use cases I just have a kernel, and an init/application binary that I statically compile into the kernel. A simple makefile gets the job done and is easier to review than the whole of buildroot.

what happens if the program crashes? with no init there is no way to restart it?

> And yeah, threadrippers are a must.

I mean, they give a wonderfully higher quality of life for compiling but they are $2,500-6,000 just for the CPU. Of course this can be done on much lower-cost CPUs! A 12-core Ryzen is already at the "I have lots of money to spend" end of what I'd suggest for something like this.

If you are compiling Linux kernels for a living, odds are near 100% your time waiting on compiles costs more than an employer buying you a beefy CPU.

Sorry, by "something like this" I meant LFS. Obviously the word "this" was ambiguous as GP said "I compile linux kernels for work".

Sounds like a valuable formative experience.

Sort of like launching a spaceship in factorio.

Same, though a bit later, circa 2007. It's great for software developers too. It's a good quick intro to many different build systems & philosophies, and shows you how the different pieces of software connect together by standard paths and package management tools like pkgconfig. Being able to quickly dive into a new project, understand how to build it and hook other software into it, debug build problems, and make quick hack changes is extremely valuable for developers.

Yep me as well sometime in that era. It was very interesting and informative. I'd never use it for an everyday system obviously but was a good excercise. And I did this on my real machine before VMs were really a thing :-)

How does it compare to install Arch linux without an installer? I remember that it took me a day or two of tinkering, when I did it for the first time (circa 2009) and I learned a lot. As of today, Arch remains my distro of choice.

It's very different. With LFS you build your own distro. You bootstrap the filesystem, build your kernel and everything else from source, and there's no package manager. That means manually managing dependencies, which is a nightmare in and of itself, and something we take for granted with Linux distros.

The LFS book is quite thorough, so you technically only need to copy/paste commands, but you learn _a lot_ about Linux in the process. I highly recommend it to anyone interested in Linux. I did it once in college, and parts of BLFS IIRC, and it's one of the few memorable projects I got from my degree.

I almost did it in 2008ish. Back then my country had a severe electricity crises, and in the summer we would get alternately one hour of electricity and one hour of blackout.

So I had to try and break the LFS build into one hour chunks. Though I got through quite a bit, unfortunately, there were some multihour build steps and I never got lucky enough with multihour electricity, till I finally lost interest.

I did learn quite a bit from reading the manual though.

> I had a 1ghz, 256gb ram machine

You had a 256gb ram machine???

Ha, I wish, good catch on the typo, I can't edit, but should have been "mb"

It's been so long that thinking of ram in anything other than GB is weird.