News for hackers. Stuff that matters.
Re:Ask HN: Why the Linux Kernel doesn't have unit tests? by titzer on Friday November 25, 2022 @11:03AM (#33743617)
Unittests is not the same thing as TDD religion. At its base, it just means dividing things up into small units and testing their functionality as exhaustively as possible. It's a divide-and-conquer.

The hard part in adding unittests is deciding what a unit is, and when a unit is important enough that it should have its own battery of tests. Choosing the wrong boundaries means a lot of wasted time and effort testing things that likely won't break or change so fast that put a drag on refactoring.

I disagree that a kernel can't or shouldn't be unit tested. At the very least, it has a strong interface in the userspace system calls. Of course you should unit test the system calls. Especially because Linux's motto is not to break userspace.

The other benefit from unit testing that is overlooked is that it accelerates optimization. If you have a good set of unit tests that test all observable behaviors of a system means you can optimize the logic inside and constantly rerun those tests to make sure it's not broken. This speeds up the experimentation process and clearly delineates the interface so that you can see when and how you can "cheat" on things that aren't observable.

So, hard disagree. Test the bejesus out of the kernel. It will harden it and open up the ability hyperoptimize the units inside.

edit to add: well-written (i.e. terse and readable) tests are excellent documentation on how a unit should behave.

Re:Ask HN: Why the Linux Kernel doesn't have unit tests? by nu11ptr on Friday November 25, 2022 @11:12AM (#33743732)
I both agree and disagree. There are PARTS of the kernel that probably should be unit tested. As I specified in my answer, these tend to algorithmic. I would also agree that an interface that cannot change ("don't break user space!") makes a good candidate possibly.

However, I think there are parts that probably should NOT be unit tested, because unit testing isn't free and slows down change by getting very close to internals. As an example, device drivers that are tied tightly to hardware. While it would sound nice to have an nvidia GPU fake/mock, in reality it is probably not possible to create one, keep it up to date, etc. The complexity would be enormous and might not show anything as you wouldn't know if it was a bug in the fake or driver.

As such, I retain my answer of: it depends what you are trying to accomplish. Different testing strategies for different types of code and domains.

Re:Ask HN: Why the Linux Kernel doesn't have unit tests? by titzer on Friday November 25, 2022 @11:18AM (#33743811)
There are some gray areas, sure. I usually find a way around making mocks, but maybe it's just the domains I am in. My biggest mistakes are mostly just writing tests that are too intertwined with internal interfaces. So they break all the time and need to be adjusted when refactoring. That's a drag.
Re:Ask HN: Why the Linux Kernel doesn't have unit tests? by thayne on Friday November 25, 2022 @11:11PM (#33749835)
I think the same is true of any sufficiently complex project. There are areas where unit tests are valuable, and areas where the costs outweigh the benefits. There isn't anything really special about a kernel in that regard (except for the specific things that aren't worth unit testing).
Re:Ask HN: Why the Linux Kernel doesn't have unit tests? by closeparen on Saturday November 26, 2022 @03:47PM (#33756561)
In a commercial setting, unit test coverage inevitably becomes a KPI that needs to be driven upwards regardless of whether rank and file engineers consider it worthwhile in particular cases.
Re:Ask HN: Why the Linux Kernel doesn't have unit tests? by xg15 on Saturday November 26, 2022 @03:53AM (#33751120)
> As an example, device drivers that are tied tightly to hardware. While it would sound nice to have an nvidia GPU fake/mock, in reality it is probably not possible to create one, keep it up to date, etc. The complexity would be enormous and might not show anything as you wouldn't know if it was a bug in the fake or driver.

Ok, but then again: How would you make sure your Nvidia driver is working correctly?

Re:Ask HN: Why the Linux Kernel doesn't have unit tests? by hnfong on Friday November 25, 2022 @12:47PM (#33744921)
Your comment sounds like TDD religion to me.

"Deciding what a unit is" isn't the hard part. The hard part is finding the units that benefit from unit tests (and convincing other religious people about this, as I am trying to now, which in this case - not many).

The Linux system calls are "deceptively simple". Simple to test, right? How complicated can say `write(2)` be? But if you actually tried doing it, I'd be surprised if you can write reliable "unit" tests beyond writing to /dev/null.

The practical way to test system calls are to test it with real world usage. Unit tests here might catch some problems, but the vast majority of kernel bugs aren't those that can be caught with unit tests. (If they're bad enough, the bug will cause the OS to crash before it completes booting, no unit tests needed.) The more subtle bugs are often hard to reproduce, only happens in certain loads and hardware.

As to your final edit-to-add: you're joking right? Like Linux syscalls need more documentation. POSIX (I'm aware this is not exactly Linux documentation, but still) was a spec literally before Linux was started.

edit PS: https://tenor.com/view/unittest-unit-test-gif-10813141

Re:Ask HN: Why the Linux Kernel doesn't have unit tests? by makapuf on Friday November 25, 2022 @01:10PM (#33745160)
I don't know how many LOC you could trigger with a write. Including all drivers, filesystems , ..., perhaps millions. Hardly a unit test. And then, a driver is difficult to unit test. BUT some parts of the kernel could use unit tests. The thing is, they don't and the usage of the kernel shows that they would be triggered by usage, but it wouldn't harm.
Re:Ask HN: Why the Linux Kernel doesn't have unit tests? by Izkata on Friday November 25, 2022 @01:55PM (#33745601)
> Your comment sounds like TDD religion to me.

GP is using the original definition of unit test, GGP is using the religious implementation-level definition TDD advocates have come up with.

From GGP:

> but integration and behaviour tests for the external interface/API are more important than exercising an internal implementation detail.

They call this an integration test, but by the original definition, testing the interface is a unit test. An integration test is how multiple units interact with each other. Testing the implementation details is what TDD advocates have turned unit testing into, by changing the definition of "unit" from semantics (a self-contained module that does one thing from the business perspective) to syntax (a function).

> edit PS: https://tenor.com/view/unittest-unit-test-gif-10813141

Seems like you agree with the original definition. The TDD version of this GIF would be a test for each part that makes up the handle (the implementation details instead of the interface).

Re:Ask HN: Why the Linux Kernel doesn't have unit tests? by hnfong on Friday November 25, 2022 @02:26PM (#33745905)
The semantics doesn't really matter either way. The comment I originally replied to basically implied the whole Linux kernel is one "unit" (or at least each kernel syscall is one unit... which isn't much difference due to large interdependencies between them).

Using that definition, that's just basically saying "the kernel can benefit from ... tests". Of course the Linux kernel doesn't have "TDD-religion unit tests", but there's already more than enough "tests" for the Linux kernel, and there's more than enough benchmark tests for the Linux kernel (for the optimization argument). They're just not "unit" tests in any meaningful way.

From the pedantic side of things, the original comment said "(TDD-religion) unit tests are overrated", then the reply was either "no they're not", or "no, unit tests (as originally defined) are not overrated". The latter is somewhat inconsistent with the "hard disagree", so either they were confused as to which definition they wanted, or cherry picked properties from both unit testing regimes.

Re:Ask HN: Why the Linux Kernel doesn't have unit tests? by narag on Friday November 25, 2022 @01:35PM (#33745383)
So, hard disagree. Test the bejesus out of the kernel.

Maybe the people that has created such a successful 31 y.o. piece of software deserve that the default option is to trust they have some reason to do it the way they do it. Just saying.

Re:Ask HN: Why the Linux Kernel doesn't have unit tests? by hbrn on Friday November 25, 2022 @05:05PM (#33747320)
> If you have a good set of unit tests that test all observable behaviors of a system means you can optimize the logic inside and constantly rerun those tests to make sure it's not broken

This is a naïve utopian world view. The word "good" is doing all the heavy lifting for you. In the last 15 years I haven't seen a single company that had "good" unit tests. I'm inclined to believe they don't exist.

Some of the tests I saw turned out to be good. Most of them weren't.

The reality is that if you're committing to unit tests, a big chunk of your tests will be shit. And when that's the case, accelerated optimization is far from guaranteed.