You don't necessarily have to be a kernel hacker to be familiar with many of the features that the kernel provides. Just doing application debugging often requires to dig deeper until you hit some kernel balrogs.

Container problems? Namespaces, Cgroups, ...

Network problems? Netfilter, tc, lots of sysctl knobs, tcp algorithms (cue 1287947th thread on nagle/delayed acks/cork)

Slow disk IO? Now you need to read up on syscalls and maybe find more efficient uses. Copy_file_range doesn't work as expected? Suddenly you're reading kernel release notes or source code.

> How do people become linux kernel hackers?

Honestly, by hacking it.

There's a famous book about Linux internals that I don't remember the name (but has "Linux" and "internals" on it). But I have never seen anybody doing it by reading a book (despite how excellent it can be). You just go change what you want or read the submodule you are interested in understanding, and use the book, site or whatever when you have a problem.

>There's a famous book about Linux internals that I don't remember the name (but has "Linux" and "internals" on it)

this one? https://0xax.gitbooks.io/linux-insides/content/index.html

though it says insides instead of internals

Yes, this one. Thanks.

I have never run into a problem that I thought needed to be solved in the kernel. What kinds of things have you wanted to change?

The first time I went into it was to write a driver for a device in my undergrad. After that I've changed a driver here or there (never anything worth merging), and needed the documentation of the sound systems.

It's not an easy thing, by any means. Just locating where you have to touch on the source tree is a problem that will lead you to plenty of books or sites. But don't try reading those before you have a problem to solve, you will lose time and drown in information.

(By the way, I am assuming you know how syscalls work. If you don't, go study that before you start anything.)

I learned 90% of the things I know about operating systems by solving problems I myself caused.

Once I found myself reading about device module programming since the common USB-Serial device module (I forgot its name, cdc something) wasn't properly working for a Chinese multiserial port chip inside a GSM cluster modem (one USB device to multiple serial ports).

I was attempting to hack away a simple example but I found the USB-Serial (more) generic driver intended for "test only" and... it just worked.

Another reason for reading about IO calls, schedulers, etc? That "I'm still writing data into your USB flash drive even when the GUI says it finished 5 minutes ago" that I hate so much.

Conversely every problem I’ve run into in a kernel that I could potentially solve requires me to be an expert in 1-3 things outside of the kernel.

Apart from Linux hw support for things at work, I implemented a fairly simple pseudo-device for establishing TCP connections from a process in capability mode on FreeBSD. The device driver has support for a denylist to disallow connections to specific IP ranges. It has multiple syscalls wrapped into one ioctl, and sockets opened from the device always had TCP_NODELAY, O_CLOEXEC and SOCK_NONBLOCK set. Worked pretty well for its intended use case.

https://github.com/sebcat/yans/blob/master/drivers/freebsd/t... https://github.com/sebcat/yans/blob/master/drivers/freebsd/t...

In my case, which is probably typical, there was a bug in a device driver for some obscure thing we were using where I worked. So I had to dive into the world of kernel modules and fix it. I think a lot of kernel knowledge and development is commercially-driven, in this sense.

I think you were thinking “Linux Core Kernel” by Scott Maxwell. And yes, it’s an awesome book that’s copied the style of a same type of book that annotates the SVR4 kernel

For the most part, it's just software. If you have the time and the interest, you can learn it like anything else. At some level, it requires an awareness of how the hardware works(page tables/MMUs/IOMMUs, interrupts, SMP, NUMA, etc).

I don't mean to downplay the investment, but if you're already an experienced software engineer you can get into it if it interests you. There is a different mindset among systems software programmers though. Reliability comes first, performance and functionality come second. It's a world away from hacking python scripts that only need to run once to perform their function.

I learned a TON about the Linux kernel through writing custom device drivers for FPGAs. Granted most of my experience is in the driver area and not in any of the subsystems, but even still I have a much better grasp of how the kernel operates now (and even more importantly, I know how to navigate it and how to find relevant documentation).

As others have said, hacking it, certainly. But if you're not up for that and would like something more passive, read LWN.net (and possibly subscribe!)

I learned a lot by trying to make Go talk to ALSA without using any existing C interfaces. Just happy exploration goes a long ways!

UTSL