News for hackers. Stuff that matters.
Re:macOS extensions are moving away from the kernel by nemothekid on Wednesday July 07, 2021 @03:37AM (#27758426)
This article does a good job of explaining how kernel extensions worked under Intel and how they will work under M1, and the blessed replacement for kernel extensions by Apple under M1. What it's missing, and what I assume everyone's questions is: Are the capabilities of Apple's user-level APIs sufficient to replace most legitimate uses of kernel extensions?
Re:macOS extensions are moving away from the kernel by pjmlp on Wednesday July 07, 2021 @03:52AM (#27758503)
The roadmap as presented by Apple at WWDC 2019 was that after an userspace replacement for a kernel subsystem gets presented, there is one year deprecation until it gets removed from the kernel on the following OS release.

So there is one year to try out what might still be missing and give respective feedback.

The long term roadmap is that macOS will eventually be much more microkernel like.

Re:macOS extensions are moving away from the kernel by AshamedCaptain on Wednesday July 07, 2021 @05:15AM (#27759049)
I am quite sure that we have been other this several times in HN.

The fact that they're no longer going to allow loading of 3rd party kernel modules (a pure political decision) does not a make a microkernel (which is an architectural style).

The kernel is still going to be big and monolithic (it contains the entire BSD kernel in the same address space!), and they're still going to be loading Apple-authorized kernel modules (500MiB of them, according to the article) in the same address space.

Re:macOS extensions are moving away from the kernel by Others on Wednesday July 07, 2021 @12:20PM (#27762842)
How is removing 3rd party kernel modules a “pure political decision”. Do you really believe there are no technical/security factors behind this choice?
Re:macOS extensions are moving away from the kernel by AshamedCaptain on Wednesday July 07, 2021 @06:10PM (#27766302)
What I mean is that it involves a policy setting, but little to no technical change. You cannot create a microkernel just because you declare 3rd party extensions to be banned. In order to classify yourself as a certain architecture you actually need that architecture first.
Re:macOS extensions are moving away from the kernel by jjtheblunt on Wednesday July 07, 2021 @12:48PM (#27763163)
It's an engineering decision : buggy kernel extensions bring down the OS.
Re:macOS extensions are moving away from the kernel by mbreese on Wednesday July 07, 2021 @01:50PM (#27763865)
Agreed. Given the number of times I personally reported a buggy (IT mandated) firewall for crashing my system, I am extremely happy that they moved extensions out of the kernel. The extension makers who wrote poor quality code have only themselves to blame for the situation.

Using a USB network adapter (requires for Ethernet on a MacBook) would bring down my system every time the firewall was running. And the stacktraces all included their calls.

I’m sure Apple, like me, got tired of getting reports about a few known bad actors.

Re:macOS extensions are moving away from the kernel by pjmlp on Wednesday July 07, 2021 @05:23AM (#27759111)
As you wish, I am no longer going to debate the pedantic meaning of pure microkernel ideology.

I rather be happy for the improvements in the long term roadmap and overall outcome for the eco-system.

Re:macOS extensions are moving away from the kernel by AshamedCaptain on Wednesday July 07, 2021 @05:40AM (#27759209)
"Pedantic"?

If you just consider disallowing 3rd party modules to be a "microkernel" all while the kernel itself still loads most if not all (approved) drivers such as PCI bus, storage devices or filesystems in the same address space, then _all_ current major operating systems are "microkernels".

Re:macOS extensions are moving away from the kernel by pjmlp on Wednesday July 07, 2021 @07:34AM (#27759859)
From all pratical purposes for a third party driver writer persona, the only path forward are user space drivers.

What you call a kernel like that?

Re:macOS extensions are moving away from the kernel by Bjartr on Wednesday July 07, 2021 @08:46AM (#27760469)
Non-extensible?

Locked?

"A kernel that only allows user-space third party drivers"?

That this style doesn't have a nifty name isn't a good reason to weaken the established definition of microkernel just so it does.

Re:macOS extensions are moving away from the kernel by pjmlp on Wednesday July 07, 2021 @09:39AM (#27760979)
It is kind of hard to implement a mikrokernel without any kind of drivers on the kernel.

So, then it becomes a balancing act how many are allowed the priviledge to be part of the main act.

Re:macOS extensions are moving away from the kernel by AshamedCaptain on Wednesday July 07, 2021 @09:57AM (#27761165)
> It is kind of hard to implement a mikrokernel without any kind of drivers on the kernel.

Well, that is kinda the point of a microkernel (or at least one of the points).

Sure, you're not going to have a kernel whatsoever without say an CPU/MMU driver in the same address space. Maybe a driver for the timer.

But when you have graphics, sound, storage, USB controllers, filesystems, networking and most of a BSD kernel (providing all the syscalls) in the same address space, now that is not a microkernel by any definition of the word.

XNU is called an "hybrid" kernel in part because in the original design the BSD parts implementing most of the actual system could have been spun off as a separate process (similar to Windows NT's original design), which would have been an actual microkernel, but in practice they were just stuck together in the same process, making it a monolithic kernel in all but name (just like in Windows).

> So, then it becomes a balancing act how many are allowed the priviledge to be part of the main act.

Yes.

Re:macOS extensions are moving away from the kernel by Nursie on Wednesday July 07, 2021 @07:36AM (#27759874)
Tivoised?
Re:macOS extensions are moving away from the kernel by ksec on Wednesday July 07, 2021 @09:05AM (#27760665)
I was under the impression Apple intend to move those ( including their own ) outside as well with NetworkKit and DriverKit?

Which is what pjmlp implies, once they are out, isn't it microkernel?

Re:macOS extensions are moving away from the kernel by AshamedCaptain on Wednesday July 07, 2021 @09:54AM (#27761141)
But they do not plan to move their own out, which is the point. That would require significant rearchitecture.
Re:macOS extensions are moving away from the kernel by robertlagrant on Wednesday July 07, 2021 @06:43AM (#27759513)
> As you wish, I am no longer going to debate the pedantic meaning of pure microkernel ideology.

This isn't pedantic, nor is it ideology. If you're acquiescing, do it graciously and please refrain from name calling.

Re:macOS extensions are moving away from the kernel by pjmlp on Wednesday July 07, 2021 @07:34AM (#27759857)
What a sensible world we live on nowadays, that wasn't name calling in any sense.

From all purposes from a third party driver writer persona, the only path forward are user space drivers.

What you call a kernel like that?

Re:macOS extensions are moving away from the kernel by robertlagrant on Wednesday July 07, 2021 @07:36AM (#27759869)
> What a sensible world we live on nowadays

This is basically calling me silly, just as you were calling the previous person ideological and a pedant. If you cut out this commentary that would be great.

Re:macOS extensions are moving away from the kernel by pjmlp on Wednesday July 07, 2021 @08:39AM (#27760404)
You are the one playing the victim here, I am not attacking anyone.
Re:macOS extensions are moving away from the kernel by robertlagrant on Thursday July 08, 2021 @05:54AM (#27769994)
I don't feel victimised; I didn't mention you attacking anyone. I'm just requesting removing the noise to boost the signal.
Re:macOS extensions are moving away from the kernel by tinus_hn on Wednesday July 07, 2021 @06:09AM (#27759359)
It’s not about pedantry; whatever rules you define for a micro kernel it shouldn’t be rules for thee but not for me
Re:macOS extensions are moving away from the kernel by pjmlp on Wednesday July 07, 2021 @07:33AM (#27759853)
From all purposes from a third party driver writer persona, the only path forward are user space drivers.

What you call a kernel like that?

Re:macOS extensions are moving away from the kernel by vimacs2 on Wednesday July 07, 2021 @11:58AM (#27762572)
A restrictive monolith kernel?
Re:macOS extensions are moving away from the kernel by jen20 on Wednesday July 07, 2021 @09:02AM (#27760623)
> The kernel is still going to be big and monolithic (it contains the entire BSD kernel in the same address space!)

macOS does not use “the” BSD kernel (it’s not even obvious what your assertion even means since there are many members of the BSD family, each with their own kernel - none of which macOS uses).

Re:macOS extensions are moving away from the kernel by AlotOfReading on Wednesday July 07, 2021 @10:03AM (#27761232)
I'd assume they're talking about XNU, which ultimately derives from "the" original Berkeley distribution like all of the other BSDs.
Re:macOS extensions are moving away from the kernel by tyingq on Wednesday July 07, 2021 @12:40PM (#27763076)
Large parts of XNU come from both 4.3BSD and FreeBSD.
Re:macOS extensions are moving away from the kernel by cpach on Wednesday July 07, 2021 @03:58AM (#27758548)
Cool! Very interesting development given how bad reputation microkernels had for a while. (I’m thinking of Linux vs Minix/Hurd.)
Re:macOS extensions are moving away from the kernel by zibzab on Wednesday July 07, 2021 @04:14AM (#27758663)
FYI, Linux userspace drivers were introduced in 2006-2007 and improved multiple times over the years. Some of my filesystem & networking drivers run in usersapce.

The monolithic vs microkernel debate was pretty clueless at times. Core Linux is smaller than dito hurd when configured with similar functionality.

OSX mach has always been a hybrid kernel. In a sense Linux is too but you can also configure it as a mostly-monolithic thing, and most seem to prefer that.

Re:macOS extensions are moving away from the kernel by creshal on Wednesday July 07, 2021 @04:26AM (#27758727)
Some of the userspace drivers are even shared between OSX and Linux, via FUSE.
Re:macOS extensions are moving away from the kernel by roblabla on Wednesday July 07, 2021 @06:03AM (#27759335)
Another example would be userspace network drivers that directly talk to the network card. Those tend to be much more performant than using the kernel driver due to avoiding the need to context switch from userspace to kernelspace.
Re:macOS extensions are moving away from the kernel by MayeulC on Wednesday July 07, 2021 @12:47PM (#27763151)
Aren't we making userspace/kernel space communication a lot more efficient with io-uring, though?

Likewise, eBPF allows userspace programs to perform some work inside kernel space to reduce syscalls.

I wonder if io-uring won't be a starting point for a "microkernelization" of Linux. It seems really effective as a message-passing interface.

Re:macOS extensions are moving away from the kernel by pjmlp on Thursday July 08, 2021 @03:22AM (#27769169)
Look at Android' Project Treble, that is what Google has done, just with Android IPC instead of io-uring.
Re:macOS extensions are moving away from the kernel by zibzab on Wednesday July 07, 2021 @10:45AM (#27761689)
Is that really true in pratice (excluding unikernels, of course)?
Re:macOS extensions are moving away from the kernel by pjmlp on Wednesday July 07, 2021 @04:55AM (#27758898)
Since Android 8, Linux is a kind of pseudo-microkernel, as all drivers are required to be Treble based, and traditional Linux drivers are considered legacy on the platform.

https://source.android.com/devices/architecture/hal-types

The Switch (yes the games console) uses a microkernel.

QNX an OS for human critical scenarios is a microkernel.

Finally, when arguing Linux vs Minix/Hurd, think of the irony of running Linux on top of a cloud hypervisor, with an endless load of Kubernetes or Docker loads.

Re:macOS extensions are moving away from the kernel by SlowRobotAhead on Wednesday July 07, 2021 @11:49PM (#27768286)
> QNX an OS for human critical scenarios is a microkernel.

Hmm, Never thought about a stated OS focus like that. Off to look it up. Good mention

Re:macOS extensions are moving away from the kernel by pjmlp on Thursday July 08, 2021 @03:23AM (#27769175)
Search for high integrity computing, that is the industry term.
Re:macOS extensions are moving away from the kernel by kzrdude on Wednesday July 07, 2021 @04:05AM (#27758603)
macOS (Mac OS X lineage) was always based on mach (a microkernel) which made it a hybrid kernel: https://en.wikipedia.org/wiki/XNU
Re:macOS extensions are moving away from the kernel by tzs on Wednesday July 07, 2021 @09:56AM (#27761160)
At the time of the Linux vs. Minix debate, on pretty much every performance measure that Linus said monolithic kernels were better at the microkernel QNX was totally wiping the floor with Linux.
Re:macOS extensions are moving away from the kernel by AshamedCaptain on Wednesday July 07, 2021 @05:16AM (#27759066)
The entire "ring 1" thing is highly likely false. It's been decades since I last saw anyone using the x86 ring levels and I seriously doubt XNU is using them.

In any case, extensions run in the same address space as the rest of the kernel making any such "rings" pointless.

Re:macOS extensions are moving away from the kernel by kmeisthax on Wednesday July 07, 2021 @10:16AM (#27761375)
The funny thing is that the M1 has an undocumented but totally used feature called Guarded Execution[0] that lets you separate parts of the kernel (or hypervisor, if you're in EL2) from one another. Basically, on the M1 each Exception Level now has a regular and a guarded sublevel, with their own page table permissions, and special instructions to enter or exit the guarded execution within the same EL.

macOS on the M1 uses this to prohibit most kernel code from touching pagetables. You have to do that from within the guarded sublevel, which kernel extensions can only call into.

This is more or less the original intent of x86 rings: to separate an OS kernel out into parts with different permissions and levels of isolation.

[0] https://blog.svenpeter.dev/posts/m1_sprr_gxf/

Re:macOS extensions are moving away from the kernel by als0 on Wednesday July 07, 2021 @06:06AM (#27759344)
Rings 1 and 2 don't even exist on AMD64/x86-64.
Re:macOS extensions are moving away from the kernel by zvrba on Wednesday July 07, 2021 @08:37AM (#27760393)
Yes they do, but segmentation is practically non-existent and page tables distinguish only between rings 0 and rings 1-3. So no, they aren't very useful, but they do exist and, AFAIK, they're still checked in "far" control transfers.
Re:macOS extensions are moving away from the kernel by tzs on Wednesday July 07, 2021 @10:09AM (#27761298)
It's too bad they have been made useless. I'm not fully sure about this, but I think that if you could have two rings for user code so that applications that want to run untrusted code in a sandbox could run the main application in ring 2 and the sandboxed code in ring 3, you could make the application safe from Spectre attacks from the sandboxed code without giving up performance.
Re:macOS extensions are moving away from the kernel by bonzini on Wednesday July 07, 2021 @01:42PM (#27763793)
The point of Spectre attacks is that they use side channel to infer the target's data from communication between the target and the attacking userspace process. This is possible even over the network, so ring 2 wouldn't help much.
Re:macOS extensions are moving away from the kernel by zinekeller on Wednesday July 07, 2021 @06:11AM (#27759371)
I mean, recent developments in the Windows/NT architecture uses virtualization to create a three-level "ring" system, so while it's not strictly architectural "rings", it's effectively a tri-ring system.

But then, I haven't delved into how macOS in M1 precisely works.

Re:macOS extensions are moving away from the kernel by AshamedCaptain on Wednesday July 07, 2021 @06:21AM (#27759413)
Virtualization is indeed the last user of the x86-ring 1 that I remember. But it is much easier to put an entire virtualized kernel in a different ring (x86 or ?) than selectively put pieces and parts of one in different rings, which no one does.
Re:macOS extensions are moving away from the kernel by zinekeller on Wednesday July 07, 2021 @06:37AM (#27759489)
> Virtualization is indeed the last user of the x86-ring 1 that I remember.

Yes, I know, but I'm referring to hardware virtualization, essentially making "ring -1" "ring 0" and "ring 0" "ring 1".

Re:macOS extensions are moving away from the kernel by mmis1000 on Wednesday July 07, 2021 @01:52PM (#27763893)
If you ever want to, it seems you can have a unlimited layer of 'rings' now? I heard that both amd and intel have nested page virtualization in recent products.

Besides that, hyperv seems make use of it to achieve paravirtulization?

Re:macOS extensions are moving away from the kernel by bonzini on Wednesday July 07, 2021 @01:43PM (#27763801)
Xen on 32-bit x86 used the real ring 1. :)
Re:macOS extensions are moving away from the kernel by smoldesu on Wednesday July 07, 2021 @11:27AM (#27762184)
> Are the capabilities of Apple's user-level APIs sufficient to replace most legitimate uses of kernel extensions?

I would have trusted literally anyone else to do a good job at this. Apple's in the business of brokering power though, I see no reason why they'd be motivated to make those APIs as rich as kexts.

Re:macOS extensions are moving away from the kernel by Someone on Wednesday July 07, 2021 @05:03PM (#27765702)
The whole reason for this change is to limit the damage drivers can do.

Because of that, nobody would be stupid enough to make those APIs (in the wide sense, as in “what kexts can do, not what they are supposed to do) as rich as kexts.

Re:macOS extensions are moving away from the kernel by Razengan on Wednesday July 07, 2021 @12:48PM (#27763158)
Why?
Re:macOS extensions are moving away from the kernel by webmobdev on Wednesday July 07, 2021 @05:15AM (#27759051)
> Are the capabilities of Apple's user-level APIs sufficient to replace most legitimate uses of kernel extensions?

From what I remember reading from some macOS Application Firewall makers, no. (And that's ofcourse, intentional.)

While it is true that poorly coded Kernel extensions can make an OS unstable, this is just another example of Apple taking away more control from its users and further crippling macOS to make it more like ios.

Re:macOS extensions are moving away from the kernel by leokennis on Wednesday July 07, 2021 @09:52AM (#27761117)
> While it is true that poorly coded Kernel extensions can make an OS unstable, this is just another example of Apple taking away more control from its users and further crippling macOS to make it more like ios.

It isn't. They are making it (very) less likely for average simpleton users like myself to expose or fuck up their system.

And for the more elite hackers such as yourself who absolutely want to load third party kernel extensions, they require you to disable SIP. The knowledge and work it takes to do that is a very nice "I hereby declare I know what I'm doing and I alone bear the responsibility if things go wrong" entry barrier.

Re:macOS extensions are moving away from the kernel by least on Wednesday July 07, 2021 @05:51AM (#27759276)
After feedback from aforementioned firewall developers, changes were made to network extensions that allowed them to do what they wanted with their applications (for the most part, at least).

I still think that it’s a worse in-practice system than kexts but they are responding to feedback at least.

You can still load kernel extensions in Big Sur by disabling SIP, though.

Re:macOS extensions are moving away from the kernel by no_time on Thursday July 08, 2021 @07:20AM (#27770535)
I wonder how far away are we from governments mandating SIP and equivalents to be turned on to connect to the internet.
Re:macOS extensions are moving away from the kernel by IfOnlyYouKnew on Wednesday July 07, 2021 @07:02AM (#27759624)
Well, I'm glad to see even the cynics can't find an evil motive here, and has to resort to just assuming stupidity.

That, however, isn't easy to do, considering Apple has a rather impressive track record of creating products people enjoy and buy.

And to jump two replies ahead: them trying to make it "easy" and "save" doesn't imply their users are idiots. Quite a few developers and scientists use Macs, and the lawyers aren't exactly dumb either. Spending hours trying to somehow get both sound and bluetooth to work at the same time, a favourite pastime on Linux, means lifetime wasted for something that shouldn't need doing. And considering nobody reads the source, there is no reason to believe some "expert" has some ability to avoid installing that one extension that soon starts encrypting their files. At least they tend to have better backups...

Re:macOS extensions are moving away from the kernel by Hackbraten on Wednesday July 07, 2021 @08:07AM (#27760149)
Ironically, kexts allow me to have better backups in the first place. APFS won’t check for bit rot, nor does it allow to make snapshots. OpenZFS enables both, and works with a kext. I don’t know whether Apple will allow that use case in the future.

I don’t like the way this is going.

Re:macOS extensions are moving away from the kernel by yehaaa on Wednesday July 07, 2021 @09:53AM (#27761131)
You can definitely make snapshots on APFS. It also use checksums for filesystem integrity.
Re:macOS extensions are moving away from the kernel by Hackbraten on Wednesday July 07, 2021 @12:17PM (#27762807)
> You can definitely make snapshots on APFS.

Apple can. I can’t. Making APFS snapshots requires special entitlements, which they’re not going to give me.

> It also use checksums for filesystem integrity.

It doesn’t check integrity of the actual data though. But I care about my data.

Re:macOS extensions are moving away from the kernel by CharlesW on Wednesday July 07, 2021 @04:12PM (#27765267)
> Apple can. I can’t. Making APFS snapshots requires special entitlements, which they’re not going to give me.

What makes you think that? Carbon Copy Cloner does this, and is made by a small independent developer.

https://bombich.com/kb/ccc6/leveraging-snapshots-on-apfs-vol...

Re:macOS extensions are moving away from the kernel by Hackbraten on Wednesday July 07, 2021 @05:13PM (#27765789)
They have managed to obtain the special entitlement.

Source: https://bombich.com/blog/2018/03/30/building-better-backups-...

They’re one of two apps in the world who have been granted the entitlement as far as I know. The other one is Time Machine.

Re:macOS extensions are moving away from the kernel by yehaaa on Wednesday July 07, 2021 @01:20PM (#27763524)
You can use tmutil to create a snapshot.

Or do you want to create an app that performs a snapshot via some API.

Re:macOS extensions are moving away from the kernel by Hackbraten on Wednesday July 07, 2021 @02:51PM (#27764514)
One does not simply create an APFS snapshot using tmutil. Using tmutil means Time Machine creates a snapshot for me. It also conveniently earmarks the snapshot for later removal at Time Machine’s own discretion.

That’s not what I want. I want to have a say in snapshot retention, not have some tool make the decision for me.

Specifically, I need precise control over snapshot retention so I can maintain several off-site replicas of my backup.

Re:macOS extensions are moving away from the kernel by EricE on Wednesday July 07, 2021 @02:22PM (#27764220)
APFS does NOT checksum the same as ZFS. APFS relies on the SSD controller to checksum/safeguard data. Which is fine so long as all your data is on SSD :p
Re:macOS extensions are moving away from the kernel by deregulateMed on Wednesday July 07, 2021 @07:13AM (#27759699)
"blessed" replacement

Why did you use the word Blessed?

I'm curious what made your brain choose that word instead of something neutral.

Reminds me of "Walled garden" marketing speak instead of "Walled Prison".

Edit- is there some sensitive nerve I hit?

Re:macOS extensions are moving away from the kernel by stefanfisk on Wednesday July 07, 2021 @07:16AM (#27759718)
I can't speak for the author, but personally I'm enough of an atheist to not read "blessed" as more than "approved by the powers that be".
Re:macOS extensions are moving away from the kernel by zepto on Wednesday July 07, 2021 @09:49AM (#27761093)
> "Walled Prison"

I'm curious what made your brain choose that word instead of something neutral.

Re:macOS extensions are moving away from the kernel by andrekandre on Wednesday July 07, 2021 @01:16PM (#27763479)
what would be a good term?
Re:macOS extensions are moving away from the kernel by zepto on Wednesday July 07, 2021 @02:16PM (#27764152)
Why do you ask?
Re:macOS extensions are moving away from the kernel by andrekandre on Thursday July 08, 2021 @07:40PM (#27778924)
i was just curious

people use metaphors with a lot of conotation (walled prison) so if there is a more neutral term, i was curious how to state it in either non-positive or non-negative way

Re:macOS extensions are moving away from the kernel by zepto on Thursday July 08, 2021 @08:36PM (#27779335)
Managed ecosystem, or even private ecosystem seem like candidates. Both seem more like factual statements rather than metaphors.
Re:macOS extensions are moving away from the kernel by andrekandre on Friday July 09, 2021 @05:55PM (#27788967)
makes sense, thanks ^^
Re:macOS extensions are moving away from the kernel by Tagbert on Wednesday July 07, 2021 @02:17PM (#27764169)
Room?
Re:macOS extensions are moving away from the kernel by rubyist5eva on Wednesday July 07, 2021 @11:33AM (#27762272)
It's a back-handed use of the word as a criticism. Apple is "god" and they "bless" things that are approved by them-and-only-them to be "good".
Re:macOS extensions are moving away from the kernel by nemothekid on Wednesday July 07, 2021 @06:06PM (#27766257)
>Why did you use the word Blessed?

It's a fairly common idiom when talking about platform APIs. Some APIs are "blessed" in that they are the APIs that they platform owners would like you to use and others they would like you to avoid. "Blessed" isn't wholly positive as it implies the platform plays favorites, and frequently the new blessed apis don't support all the old functionality.

Here is patio11 using the idiom 7 years ago:

https://news.ycombinator.com/item?id=7658396

It's not as nefarious as you are implying - I don't have an agenda, I've just been on HN for too long.

Re:macOS extensions are moving away from the kernel by andyzweb on Wednesday July 07, 2021 @08:04AM (#27760126)
I thought it might have been some kind of reference to the bless(8) command.
Re:macOS extensions are moving away from the kernel by cyberge99 on Wednesday July 07, 2021 @01:01PM (#27763325)
In context this is how it’s typically used.
Re:macOS extensions are moving away from the kernel by EricE on Wednesday July 07, 2021 @02:25PM (#27764245)
>Edit- is there some sensitive nerve I hit?

Probably the nerve that's tired of people going out of their way to be offended over terms that never were offensive when used in context such as this subject matter.

Context does matter.