News for hackers. Stuff that matters.
Re:Writing a Simple Linux Kernel Module by chatmasta on Thursday November 30, 2017 @01:06PM (#15817945)
Does anyone have a story or two of a time you’ve created a kernel module to solve a problem? I would be interested in hearing real world use cases.
Re:Writing a Simple Linux Kernel Module by mbrumlow on Thursday November 30, 2017 @01:41PM (#15818292)
I wrote one so I could keep my job...

I work at a company that provides backups for both Linux and windows. The entire concept was around block level backups. You could just open up the block device and copy the data directly but it would quickly become out of sync by the time you finished copying it. We did not want to require LVM to be able to utilize snapshots to solve the sync problem. On top of that we had a strong requirement of being able to delete data from the backup.

This resulted in me learning how to build a kernel module and then slowly over about 6 months creating a kernel driver that allowed us to take point in time snapshots of any mounted block device with any fs sitting on top.

Other requirements also dictated that we keep track of every block that changed on the target block device after the initial backup (or full block scan, after reboot).

I wish I could release the source but my employers would not like that :( So at least for me learning how to write kernel modules and digging in to some of the lower stuff has keep me gainfully employed over the years. It is still in use on about 250k to 300k servers today (it fluctuates).

The hardest part was not writing the module, but getting others interested in it enough so I don't have to be the sole maintainer. I like working on all parts of the product and don't want to just be the "kernel guy".

One other time I wrote a very poorly done network block device driver in about 8 hours. You can find it here https://github.com/mbrumlow/nbs -- Note I am not proud of this code, it was something I did really quick, wanted it on hand to show to a perspective employer -- I did not get the job, I am also fairly sure they did not even look at the driver, so I don't think the crappy code there affected me.

Re:Writing a Simple Linux Kernel Module by throwaway613834 on Thursday November 30, 2017 @01:54PM (#15818405)
So you implemented something like Windows's volume snapshots/shadow copies? (Why couldn't you use the existing feature?)

EDIT: Thanks for both replies! :)

Re:Writing a Simple Linux Kernel Module by aseipp on Thursday November 30, 2017 @02:06PM (#15818529)
Pretty much. But Windows snapshots aren't writeable after you take them, which is a major downer and a major necessary feature. (Having this feature lets people do things like "Take snapshot and delete everything except *.lic files, under C:\Users, on every backup" and you can just issue deletes to the shadow volume after it is consistently created, and intercept the new deletions, like normal. Tracking deletions like this lets you more easily know what you should/should not back up -- only back up the allocated blocks in the shadow copy.)

So, there was a thing that added this feature to the Windows kernel in the product to make this work. Aside from the Linux stuff, which was totally separate. But if you don't need the writing capability, Shadow Copies are good enough, sure..

(Source: I used to work with the guy who made the above post)

Re:Writing a Simple Linux Kernel Module by mbrumlow on Thursday November 30, 2017 @02:12PM (#15818578)
Yes, and no.

Windows Volume Shadow Copy has the advantage of being integrated with he FS a bit closer. So in Windows VSS can avoid some overhead by skipping the 'copy' part and just allocating a new block and updating the block list for the file.

For the Linux systems we had the requirement to work with all file systems (including FAT). So we could not simply modify the file system to do some fancy accounting when data in the snapshot was about to be nuked. So that resulted in me writing a module that sits between the FS and the real block driver. From there I can delay a write request long enough to ensure I can submit a read request for the same block (and wait for it to be fulfilled) before allowing the write to pass through.

> (Why couldn't you use the existing feature?) We did on Windows, VSS is used with a bit of fancy stuff added on top. For Linux there is no VSS equivalent (other than the one I wrote, and maybe something somebody working on a similar product may have written). And even if one did come about (or is and I am just not aware of) it for sure was not available when I started this project.

Re:Writing a Simple Linux Kernel Module by ploxiln on Thursday November 30, 2017 @02:43PM (#15818862)
It sounds like you implemented something very similar to the LVM snapshot feature. You didn't want to require LVM but ... is that really worse than requiring your custom module which is roughly equivalent?

EDIT: ok so it looks like the management of the snapshot space is a bit different. still, you could probably have wrapped LVM management enough to make it palatable, in less than the time it took to write a custom module

Re:Writing a Simple Linux Kernel Module by mbrumlow on Thursday November 30, 2017 @04:36PM (#15819766)
The problem is requiring LVM is a NOGO if they did not already have LVM. We could have standardized on LVM, but then all the people who were not already using LVM at the time would not be able to use the product. At the time many hosting providers -- who we sold too -- just were not using LVM. We to this day still have many fewer people using LVM than raw block devices, or some sort of raid.

Also at the time LVM snapshots were super slow. I don't have the numbers but even with the overhead my driver created I was able to have less impact on system performance.

I was able to do some fancy stuff to optimize some of the more popular file systems by making inline look-ups to the allocation map (bitmap on ext3). This allowed me to not COW blocks that were not allocated before the snapshot. This was a huge saving because most of the time on ext3 your writes will be to newly allocated blocks.

Wrapping LVM would probably not work, and still require a custom module to do, the user space tools don't do much. LVM really is a block management system that needs manage the entire block device, so existing file systems not sitting on top of LVM would get nuked if you attempted to let LVM start managing those blocks, and you still had the issue that reads and writes were coming in on a different block device. Asking people to change mount points was not a option. There were also some other requirements like block change tracking that LVM does not have the concept of doing. This is for incremental snapshots. Without this sort of tracking you will either have to checksum every block after every snapshot if you wish to to only copy the changes. This module also was responsible of reporting back to a user space daemon that keep a map of what blocks changed. So when backup time arrived we could use this list (and a few other list) to create a master list of blocks that we need to send back. This significantly cuts down on incremental backup time. Some companies call this "deduplication" but I feel that is disingenuous -- to me deduplication is on the storage side and would span across all backups.

So yes, requiring a module is much easier than telling a customer they can't trial, or use this product until they took their production system off line and reformatted it with LVM. Many people hated LVM at the time, it was considered slow and caused performance problems, this was like 8 years ago... LVM has vastly changed and does not have these type of complaints any more. But I can tell you people still are going to scream bloody murder if we told them they had to redo their production images and redeploy a fleet of 200+ servers just to switch to LVM so they could get a decent backup solution.

Also shout out to aseipp! Miss working with you. Have yet to find a bug in the code you wrote :p

Re:Writing a Simple Linux Kernel Module by alxlaz on Thursday November 30, 2017 @01:25PM (#15818137)
Most Linux device drivers are kernel modules, so I suppose most of the stories I have would be of the form "I needed to write a device driver for X" :-). Userspace drivers, while certainly in existence, are not very popular in Linux land.

Other interesting stories would include:

- Instrumenting certain filesystem operations (all modules share the same memory space; it's possible for one module to take a sneak peak at another's internal structures). This was back before dtrace & friends were a (useful) thing.

- Real-time processing of instrumentation samples. Doing it in the kernel allowed us to avoid costly back-and-forth between user and kernel memory -- but we only did relatively simple processing, such as scaling, channel muxing/demuxing and the like. If you find yourself thinking about doing kernelside programming because carrying samples to and from userspace is too expensive, you should probably review your design first.

Re:Writing a Simple Linux Kernel Module by craftyguy on Thursday November 30, 2017 @08:53PM (#15821338)
> Userspace drivers, while certainly in existence, are not very popular in Linux land

Actually, that's not true at all! Just look at Mesa3D and any number of GL proprietary blobs from graphics vendors. Userspace drivers are actually quite common, they just aren't what folks think of immediately when they think about 'drivers'.

Re:Writing a Simple Linux Kernel Module by alecdibble on Thursday November 30, 2017 @03:38PM (#15819291)
A few years ago, I worked at a company that had hardware peripherals conntected to a master CPU card running linux through RS-232. One of the peripherals was a power supply that supplied power to another peripheral. Since they were architected as Slaves, the peripherals couldn't directly talk to each other, they had to get the messages relayed through the master CPU.

This whole communication process had to happen within a small time frame, something like 50 ms. A kernel module handled the communication, decoded the packets, and automatically relayed peripheral messages to other peripherals. The kernel module made it easier to achieve stable and real-time operation within the time constraint.

When writing the same function in userspace, there was absolutely no guarantee that messages would be sent or received in time.

Re:Writing a Simple Linux Kernel Module by andrewcchen on Thursday November 30, 2017 @03:01PM (#15818984)
I modified a kernel module to use the wifi enable/disable button on my hp laptop as a normal button. For some reason the bios sends an acpi event when the button is pressed, instead of just having it as a normal keyboard key, so there is a kernel driver to handle it.
Re:Writing a Simple Linux Kernel Module by sly010 on Thursday November 30, 2017 @01:15PM (#15818046)
I didn't end up writing the kernel module, so this might be an anti use case:

I was working on an embedded system and needed to have fast I2C access (mostly just wanted very small latency bc it's an instrument). I2C from Linux userspace (using ioctls) adds a lot of overhead. I started looking into kernel modules but after a day of research I found out that you can access hardware registers from userspace using mmap("/dev/mem") which is even faster than a kernel module.

Edit:typos

Re:Writing a Simple Linux Kernel Module by zeusk on Thursday November 30, 2017 @03:32PM (#15819241)
> mmap("/dev/mem")

but that would only work if some kernel module or the kernel itself hasn't mapped that IO space, right?

Re:Writing a Simple Linux Kernel Module by sly010 on Friday December 01, 2017 @11:41AM (#15824986)
I don't think mmap stops you from mapping the same memory twice, nor does linux. In my case I didn't even disable the system I2C driver. If course if I tried to use it I would get io errors and if I am not careful I can crash the kernel, but that's what you get for poking around. This was a raspberry pi.
Re:Writing a Simple Linux Kernel Module by zeusk on Friday December 01, 2017 @04:39PM (#15827786)
Yeah, that's what I meant since i2c is a serial protocol and most drivers assume mutual exclusion on device registers.
Re:Writing a Simple Linux Kernel Module by ryandrake on Thursday November 30, 2017 @01:44PM (#15818311)
This was a super-long time ago, but a kernel driver was needed for video capture back in the 90's, to control devices across I2C, etc. At the time I had a Matrox video card which did analog capture, decoding and TV-in, and so wrote the first few iterations of a kernel driver[1] to get it to work. It since got adopted by a great maintainer. Nowadays I'd guess nobody is using it because the hardware is ancient.

1: http://www.cs.brandeis.edu/~eddie/mga4linux/

Re:Writing a Simple Linux Kernel Module by lloeki on Friday December 01, 2017 @10:10AM (#15824194)
Hardly a problem per se but I was bored on a long train trip (7+ hours) and I just happened to have (on purpose) a Xbox One controller around, so I wired it in, hacked around on userland with libusb, then wrote a kext for the thing to appear as a proper HID device on OS X.

Two lessons learned:

1. dealing with tangible things for once is incredibly satisfying.

2. next time, download the fscking docs instead of relying on an intermittent 3G connection.

https://github.com/lloeki/xbox_one_controller/

Re:Writing a Simple Linux Kernel Module by zb3 on Thursday November 30, 2017 @03:54PM (#15819421)
I wrote one so that I could set hardware breakpoints and use them to extract decrypted data from an app which was heavily protected against debugging. The thing was that the app (running as an unprivleged user) had no way of knowing what I was doing in the kernel (uprobes can be used for this too providing the code doesn't checksum itself). But I guess reverse engineering is not a common use case for writing linux kernel modules though :)
Re:Writing a Simple Linux Kernel Module by anitil on Thursday November 30, 2017 @06:58PM (#15820789)
Maybe not common, but super interesting if you want to share more.....
Re:Writing a Simple Linux Kernel Module by zb3 on Friday December 01, 2017 @03:54PM (#15827363)
I guess that kernel modules are not commonly used for RE because in general there are better tools that support emulation, symbolic execution, dynamic instrumentation and so on (such as qira, vtrace, angr, Intel Pin and more, there are many articles about them). I just wanted something simple and portable to extract/modify registers/memory pointed to by registers/unwind stack when a given instruction was executed, while being as fast and transparent as possible, so I wrote a kernel module... and while it worked(hook mmap via kprobes to get base address + register_user_hw_breakpoints + printk to get output), using it was alot of manual work, and I'm sure tools I listed above could be more efficient. Also since there was no emulation, I had to spoof opening /dev/ urandom, which you don't have to do if you're using qira

It'D be great if My small module Could be published As is, but I'd need to strip all addresses/filenames, and also add more proper locking (but yes, I want to publish it anyway).

Re:Writing a Simple Linux Kernel Module by invernomut0 on Thursday November 30, 2017 @02:28PM (#15818724)
Some years ago I was working on an embedded platform with two processors connected via ethernet, one handling data and most of the application load, the second one handling some automotive buses (CANbus and ARCnet). I wrote a kernel driver to expose raw data coming from the ARCnet to userspace applications in the most transparent way. So I labeled data coming from the second cpu with an unused ethertype, I changed the ethernet driver to inspect the ether type of the incoming data and send it up inside the network stack as if it was received from a certain fake ARCnet device. The fake ARCnet device was a dummy module that allowed userspace to receive that data via normal sockets. Probably it could be done in a better way but it was quite simple and it worked well.
Re:Writing a Simple Linux Kernel Module by samtho on Thursday November 30, 2017 @01:10PM (#15817978)
I wrote one that decodes Wiegand[0] off of GPIO. I ended up not using it because epoll-ing the pins is easier to use from a userspace perspective and fast enough to capture the transmission. I loosely based the kernel module off of one that was written for another microcontroller[1].

[0]: https://en.wikipedia.org/wiki/Wiegand_interface

[1]: https://github.com/rascalmicro/wiegand-linux

Re:Writing a Simple Linux Kernel Module by ianhowson on Friday December 01, 2017 @07:17AM (#15823381)
I did the same but kept it in-kernel to meet Wiegand timings. I had (horrible) another driver which could hang the system for long enough to lose bits.

I was in Australia for that work too...

Re:Writing a Simple Linux Kernel Module by stevekemp on Thursday November 30, 2017 @10:23PM (#15821665)
I've written simple Linux Security Modules, as learning experiences, but also to help tighten up a bunch of virtual machines.

The one that I'm most proud of delegates decisions on whether executables can be run to .. userspace. Which is simultaneously evil and brilliant.

i.e. User "foo" tries to run "/tmp/exploit" and the kernel executes "/sbin/can-exec foo /tmp/exploit". If the return code of that is zero then the execution is permitted otherwise it is denied.

This gives you ample opportunity to log all executables, perform SHA1-hash checks of contents, or deny executables to staff-members outside business hours. There's a lot of scope for site-specific things, creativity is the limit!

Re:Writing a Simple Linux Kernel Module by bincat on Saturday December 02, 2017 @07:17PM (#15835174)
I think it would be much appreciated if this software could be released as open source.

How did the user space redirection handle potentially high spikes of execve's?

And btw, thanks for all the years of debian-administration.org.

Re:Writing a Simple Linux Kernel Module by aseipp on Thursday November 30, 2017 @01:28PM (#15818171)
Yes, I wrote one a long time ago that exposed basic counters from the ARMv7 PMU to userspace -- where they are normally not accessible unless used through perf_events. This is because perf wasn't available (I think the shitty old vendor BSP I was using couldn't support it for some reason) and it's relatively costly (perf_event_open is a whole syscall + another syscall to `read` off the event info) -- I just wanted some basic cycle timings for some cryptography code I wrote to see what worked and what didn't.

Even though it has some annoying gotchas (such as the fact ARM cores can sleep/frequency scale on demand with no forewarning, meaning cycles aren't always the most precise units of measurement), and is very simple -- this thing ended up being mildly popular. Even though I wrote it years ago, someone from CERN recently emailed me to say they happily used it for work, and someone from Samsung ported it to ARMv8 for me...

(I should dust off my boards one of these days and clean it up again, perhaps! People still email me about it.)

Re:Writing a Simple Linux Kernel Module by megous on Thursday November 30, 2017 @01:15PM (#15818047)
Usually to control on-SoC peripherals/IP blocks or to add support for other external devices (that's what kernel's for, mostly, anyway) - camera sensor, I2C controlled regulator, thermal sensor,... story? no. :D I just wanted to have a nice non-amd64 SoC based desktop, and some things were missing at the time.
Re:Writing a Simple Linux Kernel Module by souprock on Saturday December 02, 2017 @12:40AM (#15830389)
My real-world use case is a hypervisor. That turns out to be some pretty wild code. Intel's VMX documentation is not well-organized or complete, processor models vary, the hardware has bugs, and some things we have to do aren't even officially supported by Intel. We have to mess with page tables. We have a huge block of assembly code. It all has to run perfectly.

In case that doesn't make you want to run away screaming, I put a post in the "Who is hiring?" article:

https://news.ycombinator.com/item?id=15829518

Re:Writing a Simple Linux Kernel Module by woodrowbarlow on Thursday November 30, 2017 @04:05PM (#15819529)
i was working on an embedded device with only RAM and a flash chip for the software blob.

i wrote a kernel module that allowed us to keep logs in RAM even across a soft reboot (i.e., if the device crashes or does a software update). it basically reserved a chunk of 1M at the top of RAM (using the same physical address each time). there was a checksum system that allowed it to tell during boot whether the data is still valid or if the bits had decayed.

i've also done a couple i2c drivers for temperature sensors or LED controllers. kernel work is fun.

Re:Writing a Simple Linux Kernel Module by ship_it on Friday December 01, 2017 @09:23AM (#15823951)
Not for Linux per se but I've been developing a variant of CAN(+FD) kernel extension for XNU/OSX. Some things are manageable through userland so not everything is done in kernel but extending berkeley sockets is really fun and I'd suggest anyone interested to try it.