The code where the bug happens is legal C++, but it uses absolutely none of the memory safety improvements which were added to the language in the past... twenty years probably. It's basically C with classes.

If they haven't kept up with the changes in their current language, what makes one think that they would "move on" to the alternatives, two of which aren't even alternatives?

Before they switch to Rust it would be much faster and more efficient to use smart pointers, std::array, std::vector and stop using memcpy.

Note that this code is shipping as a kernel extension, which uses Embedded C++, not standard C++. Notably, things like templates and exceptions are not available. It would be nice if they could work on this instead, but looking at the dyld and Security sources (which has no such limitations, as the run in userspace) I don't have much confidence.

They could still make use of bounds checking, like my own classes did back in the MS-DOS days, when C++ARM was pretty much the only thing available.

Naturally when one writes C in C++ it doesn't help.

I suspect that they'll adopt MTE first.

I agree, that was the path taken by Solaris SPARC and it is the only way to make it work, because even if a language level safety would be introduced today, not everyone would bother adopting it.

As much as I like to bash security critical code written in memory-unsafe languages, I don't think that this is the crux of the problem here.

To me it's that this extremely trivial bug (the heap overflow, let's ignore the rest for now) passed through code review, security review, security audits, fuzzing... Or that Apple didn't have these in place at all. Not sure which option is worse.

We have 30 years of experience showing that ordinary heap overflows are not in fact easy to spot in code review, security review, security audits, and fuzzing. Each of those modalities eliminates a slice of the problem, and some of them --- manual review modalities --- will remove different slices every time they're applied; different test team, different bugs.

To me, this strongly suggests that the problem is in fact memory-unsafe languages, and not general engineering practices.

Apple, by the way, has all the things you're talking about in place, and in spades.

I agree that the problem is memory-unsafe languages.

You can improve the tools, or you can improve the human, and nobody has managed to improve the human despite decades of trying.

OTOH, we don't really have evidence to show that memory safety is effective in kernels/drivers because no memory safe language has ever been deployed at scale for that purpose.

The way I look at it is that relying exclusively on manual review is at best the same as relying on both manual review and a memory safe language.

In practice, the best case and average case rarely line up.

You don't have to manually review for classes of vulnerability that your programming environment forecloses on.

Good point - all the more reason to use a memory safe language!

> the problem is in fact memory-unsafe languages, and not general engineering practices.

Languages don't introduce bugs by themselves. Engineers produced those bugs.

I always thought that bugs are the programmers' fault, and not to blame the language. It's like blaming the English language because it allows you to misuse it and manufacture very offensive racial slurs, or to be rude and cruel, and thus we should replace it with another language that doesn't allow to exploit these weaknesses. We won't be able to express ourselves with beautifully (low-level) crafted poems anymore, but that's the price to pay.

There are inherent features of human languages that force you into weird issues. English for example has gender pronouns and that's why you see in profiles how you should approach someone. It's not like they want to add it, it's that they have to if it bothers them when people misuse them.

In Hungarian we don't have this problem at all, there's no concept of gender specific pronouns.

Such bugs are extremely difficult to prevent at scale. Even the most talented engineers make such mistakes and programming quality varies significantly even within top engineering teams which are usually comprised of people with different skill sets (+ junior engineers that need training).

Safe languages are the only way forward to drastically reduce the problem. It can’t be guaranteed to be eliminated 100% obviously because there are still escape hatches available, but it will be significantly improved because you can limit the surface area where such bugs can live.

Not sure if this is common for everyone but I find whenever I get assigned for a review for a monster change, I spend over an hour just working out what the change does and if it seems like it will work. There is no way I could spot tiny potential security exploits in 3000 lines of changed code.

Sending 3,000+ LoC code reviews is not considered good engineering practice. In general it's important to keep code reviews small for the reasons you describe. If it's impossible to make incremental changes in smaller units, that's a sign of an unhealthy code base.

Although it is certainly challenging, you shouldn't spend only an hour reviewing a 3,000 LoC review, unless most of it is trivial refactoring, especially if that code is security-critical and handles untrusted input. That's only 1.2s per line of code. No chance you have good quality control with that amount of attention.

If it's taken someone a whole fortnight to write it, you should expect to spend at least half a day reviewing it, IMO.

Does any software producer do fuzzing on their own product? I have never heard of this being done by software developers. Usually it's done by exploit developers. Of course there are static analysis tools that should uncover a problem like this, and I know that high-reliability embedded software developers use them, but I don't know if the likes of Apple does.

IMO this is huge.

Thankfully, Apple is starting to hire Rust developers as well as AWS.

The tide is changing, one day we will see some Rust code in iOS/macOS so that these issues are a thing of the past.

It seems unlikely that they’ll solve the problems in iOS with Rust.

It seems much more likely that they will use Swift in some form.

Swift seems an unlikely choice for incrementally replacing portions of kernel code.

Apple clearly states Swift is a systems programming language.

> Swift is a successor to both the C and Objective-C languages.

https://developer.apple.com/swift

> Swift is intended as a replacement for C-based languages (C, C++, and Objective-C).

https://swift.org/about/

There are systems languages and then there are systems languages. As the Golang team have pointed out, there's lots of systems programming going on outside of OS kernels. Neither of those links mention kernel development. Pervasive refcount updates (ARC) and a vtable-unfriendly dynamic dispatch mechanism inherited from Objective-C are fine for userspace, but most kernel code is very performance-sensitive.

Go team only changed their message due to the pitch forks they were getting from the UNIX/C crowd.

There are people that believe systems programming languages in an OS kernel is doable with some form of GC, and then there those that will never change their mind.

https://www.f-secure.com/en/consulting/foundry/usb-armory

https://blog.arduino.cc/2019/08/23/tinygo-on-arduino/

https://gvisor.dev/

https://github.com/mit-pdos/biscuit

https://github.com/ycoroneos/G.E.R.T

By the way, NeXTSTEP drivers were written in Objective-C, and the new userspace framework, DriverKit, is an homage to the NeXTSTEP driver API.

https://www.nextop.de/NeXTstep_3.3_Developer_Documentation/O...

I'm familiar with Biscuit and Midori. The Biscuit paper estimates a 10% performance hit in syscall-intensive benchmarks due to using Golang vs. C.

That being said, ARC may faire much better than a tracing GC, particularly in latency variance, throughput variance, and mean heap usage.

Though, I think we're best off moving off of hypervisors and onto 4th generation microkernels with isolation similar to AIX LPARs / Solaris Containers. After all, a hypervisor is essentially a microkernel that's forced to use hardware traps/faults plus hardware emulation for its syscall interface (plus upcalls for performance, which use a calling convention much closer to traditional syscalls). There are stability, security, and performance advantages to throwing out all of that hardware emulation code, moving everything to upcalls, and getting rid of the second (guest OS) kernel running between the application and the hardware. If you push most of the system functionality out of kernel space, then rewriting some of the less performance-critical components in a language with ARC or tracing GC starts to make more sense.

ARC is slower than tracing GC as all benchmarks where Swift was used prove.

Biscuit was a research project whose goal was to get the thesis done, when the thesis was done, no more effort was spent.

Bing was powered by Midori for the Asian countries for part of its life, and Joe Duffy has stated in the RustConf keynote that Windows Dev team was a reason why Midori faced so much internal resistance, even when they were proven wrong.

Before Google was willing to put the money into JavaScript many would assert that JavaScript would be worthless to anything besides form validation and DHTML.

Mainframes to this day still make use of their own systems programming languages, way safer than C, and you don't see people crying that their kernels are too slow.

In fact, Unisys uses this fact to sell ClearPath MCP to customers that value security above anything else, Fort Knox style.

Very interesting! I wasn't aware Midori had been widely deployed in production! My understanding is that Midori ran the kernel and all programs in ring 0 and relied on the classloader enforcing type safety and the managed runtime enforcing the other security and stability constraints normally enforced by hardware, so syscalls were just normal method calls and there were no context switches.

Burroughs MCP was written in essentially an extended Algol 60 dialect. Algol 60 only had very limited heap allocation for dynamic arrays, no GC, and I haven't read any indications that Burroughs added GC to their extended dialects.

Multics was written in a PL/I dialect, without tracing GC. Likewise, IBM OS/360 and descendants are written in the PL/S dialect of PL/I, and I haven't seen any indication it has tracing GC.

With tracing GC, you have a trade-off between the peak amount of unclaimed garbage and the GC overhead. ARC should have lower variance in both latency and heap usage, which I presume is the reason Apple moved the whole Objective-C and Swift ecosystem to ARC and deprecated the Objective-C tracing GC.

I used to be a True Believer(tm) in the JVM and other managed runtimes. I was one of 5 developers of the most popular Java desktop application in the mid 2000s. Then I moved to Google and started developing web search infrastructure. I was at Google when V8 was created, and I put a lot of effort into running all of the JavaScript that the indexing system found, across the entire visible web. For things at massive scale, spending millions of dollars per year just in electricity bills, it's extremely tough to beat highly tuned C++. Yes, it's a lot of effort. Yes, I hope safer languages like Rust replace C++ and static analysis tools continue to improve.

I still kind of want to be a managed runtime true believer again, but it's tough to go back after believing for so many years that managed runtimes were going to match expertly hand-optimized C++ in latency- and throughput-critical applications "any day now".

For me it sufices the little victories achieved by F-Secure, Aicas, PTC, Astrobe.

Fortunately there are enough true believers to keep them in business.

By the way, apparently it is time for Google to improve their gRPC C++ implementation.

https://devblogs.microsoft.com/aspnet/grpc-performance-impro...

As for the mainframe languages, yes they don't have a GC, but they have the right defaults regarding bounds checking, implicit conversions, explicit unsafe code.

Regarding managed runtimes, versus C++, languages like C#, D, Modula-3, Swift have the features to write C++ like code when needed, the main problem is that many don't bother to learn the language features available to them.

At Microsoft stories about hard core C++ devs having to be proven wrong with C# running in front of them is relatively known, Joe Duffy has shared a couple of such stories.

His experience in Singularity and Midori is also what made him bet on Go for Pulumi.

In its current form perhaps this is true. However Chris Lattner and others have expressed the desire to have it be suitable for systems programming.

There is reason to believe they will adapt it.

Has Chris specifically mentioned OS kernel programming? As the Golang team have pointed out, there's lots of systems programming going on in userspace, which is what they refer to when they call Golang a systems programming language.

That being said, ARC is probably easier to get to work well in-kernel vs. tracing garbage collection. There's a performance cost to all of those reference count updates, but at least the variance is extremely low.

By the time we all retire in a few decades they'll be a thing of the past, probably.

There's so much low-hanging fruit to pick in that code and switching to Rust is like saying that we should go to Mars to pick fruit instead.

C#? You've got to be joking.

It’s less insane than you might think:

https://en.m.wikipedia.org/wiki/Singularity_(operating_syste...

I agree rust is probably better suited. Or Apple could make their own memory safe language. They’re clearly capable.

That's an interesting experiment, but that's all it is. The project relies on ASM/C/C++ to boot into a microkernel and to interpret and run the C#. But I suppose it would greatly reduce the attack surface of C/C++/ASM code.

I just wonder, for example, how a capable hardware abstraction layer would work in C#, interrupt handling, CPU and IO scheduling, etc.

Here is another experiment, this one being delivered into production,

https://www.wildernesslabs.co/

You mean Swift?

C#, Go, and Java all go in the same category (roughly)—they wouldn't work for kernel code. Rust will be a valid replacement for C++ kernel code in the near future, I'm sure.

Here's a POSIX kernel in Go, written explicitly to prove your point wrong:

https://github.com/mit-pdos/biscuit

https://pdos.csail.mit.edu/projects/biscuit.html

Where do you get that? Those have all been used in kernels, they work.

Also on another front Apple seems to have already enabled device drivers in user space: https://developer.apple.com/system-extensions/

Yes, you can use them to play around and experiment with OS-level stuff and even have some fun with it.

But you definitely cannot build a performant and robust kernel from scrtach with these languages.

> you definitely cannot build a performant and robust kernel from scrtach with these languages.

I think this is irrelevant (and less importantly, false). Its's fine to use eg se4l as a foundation, or as a incremental step even just have a safety focused driver runtime inside the kernel proper.

Sure they would, so much that there people doing it right now.

https://www.wildernesslabs.co/

https://labs.f-secure.com/blog/tamago/

https://www.ptc.com/en/products/developer-tools/perc

Writing the majority of a kernel in those languages is certainly possible.

Possible, perhaps, but feasible? Microsoft certainly had a go at it with the likes of Midori and Singularity, but these were met with the same fate that will likely befall any managed code kernel. While it's an honorable pursuit with certain merit, to produce a fully featured OS in this way -- without serious concessions -- is just not feasible.

Those projects died due to management politics from WinDev, nothing to do with capabilities.

So much that many of the System C# features are now in .NET 5 and other ones will eventually land on .NET 6.

Nope, some people are actually quite serious about it.

https://www.wildernesslabs.co/

C# is GC'd so massive memory hit, and also not a language you can have in a kernel.

Go: GC again, so no go.

Rust: most sane of the examples you've given.

Apple has already started migrating to Swift which is a memory safe language.

However the real reasons Rust and Go aren't feasible is that they're both essentially all-or-nothing, and neither offers even the most basic semblance of ABI compatibility. Their only nod to ABI stability is "use FFI to C" which means your APIs remain unsafe, and doesn't work for non-C languages without all your system APIs having other languages layered on top.

Swift at least lets you replace individual objc classes one at a time, and is ABI stable, but has no C++ interaction.

Swift is far more like C# than Rust in terms of memory management. Sure it uses ARC but arguably that makes it not suitable for kernel level stuff.

xnu is refcounted, its also c++ which isn't swift friendly.

XNU also has ABI stability requirements which rules out rust.

Yes, but what about these huge legacy codebases like the iOS kernel? I assume we will have to deal with this type of vulnerability for years to come...

Could also fire everyone on the C/C++standards bodies and replace them with people willing to add arrays as a first class data type.

I had that argument with the C standards people a decade ago. [1] Consensus was that it would work technically but not politically. The C++ people are too deep into templates to ever get out.

The basic trick for backwards compatibility is that all arrays have sizes, but you get to specify the expression which represents the size and associate it with the array. So you don't need array descriptors and can keep many existing representations.

Also, if you have slices, you rarely need pointer arithmetic. Slices are pointer arithmetic with sane semantics.

I'm tired of seeing decade after decade of C/C++ buffer overflows. It speaks badly of software engineering as a profession.

[1] http://www.animats.com/papers/languages/safearraysforc43.pdf

Or you could be like Jonathan Blow who claims he never has any memory bug issues and so it's not a problem worth solving in his JAI language.

The political aspect is why I suggest the solution is to just up and fire all those guys. More realistically Microsoft, Apple and Linus could just force the issue. Gets added to Visual C/C++, LLVM, and Gnu C as an extention. And then start polluting code bases and API's with it.

Microsoft is already kind of doing it, if you compile in debug mode, you get bounds checking in all STL types, and you can enable them in release builds as well.

But yeah, it only works if you use those types.

There are other divisions pushing for .NET and Rust systems code in Windows, but the political wars between WinDev and DevTools are quite well known, e.g. Longhorn (in .NET) vs WinRT (same thing just in COM).

Proper thing is bounds checking should be by default and you actually have to turn it off for hot path code. Because 99% of the code people write isn't memory or CPU bound. If a little used code path is sometimes throwing bounds exceptions you want that to get logged in production.

Check C.A.Hoare Turing award speech in 1981 regarding the use of bounds checking in the 60's and how customers of Algol compilers considered a legal offence to even turn them off.

libstdc++ also has a debug mode with bounds checking etc, although you have to enable it by defining _GLIBCXX_DEBUG: https://gcc.gnu.org/onlinedocs/libstdc++/manual/debug_mode.h...

Thanks for the heads up.

I read this line:

> Gets added to Visual C/C++, LLVM, and Gnu C as an extention.

And had to do a double-take because I thought you were talking about gets(3)…

I'm not sure what exactly your are trying to say. As far as I can tell, there are indeed safe variants for arrays in the standard - both static and dynamic. People just choose to not use them for some arbitrary reasons.