> It's not just Qualcomm it's closed source driver blobs in general.

To be precise, Qualcomm's Linux drivers are GPL'd like the rest of the kernel. Drivers often come with firmware blobs and proprietary binaries in userspace, but it should be possible to keep the kernel driver compatible with both.

The actual reason why older SoCs stop working on newer Linux kernels is that Qualcom's drivers were not fully upstreamed, either because they didn't submit them, or because they tried, but there were quality issues.

In theory, anyone could port those drivers onto the latest Android kernel, but without hardware documentation it's not an easy job.

That may keep the phone working but that doesn't provide security updates. Only Qualcomm can provide security updates for the binary blobs, so if Qualcomm won't sell security updates past 3 years than no OEM can guarantee a secure phone for more than 3 years. Sure, they can update other components but I would be very upset if 3.5 years into my 5 year supported phone a CPU vulnerability was discovered that let apps get root.

So while in practice this is rare it means that longer updates can't be offered as a product feature, and if it can't be offered as a product feature than how does it help sell phones.

I'll tell you right now that if any somewhat reputable manufacturer promised 5 year support on an otherwise reasonable phone I would buy it (and 5 isn't even that much). I would even pay a bit more for it. The amount of environmental waste that is basically working phones that just need a couple of software patches is disgusting.

> I'll tell you right now that if any somewhat reputable manufacturer promised 5 year support on an otherwise reasonable phone I would buy it

I think that's exactly what Fairphone does. Sure they are not as big as Samsung, but they seem to be true to their word (I'm a happy FP3 customer, but to be honest I have much reasonable expectations for my phone than many :) )

This was one of the big reasons my newest phone is an iPhone, not an Android phone. I am tired of buying new hardware every 2-3 years just to have a secure device.

I worked on the Linux Kernel team at Qualcomm for about five years, left in 2016. Our director was very much in favor of upstreaming but the product release schedules were so aggressive everyone was scrambling just to hit customer deadlines. It was incredibly difficult to get upper management on board with spending an additional 20% time getting drivers upstreamed, despite our team making the case for it ad nauseam.

Hopefully they recognize the value of "upstream first" more now. I think it is getting better...

My understanding is that a number of Android vendors operate by just taking a version of the Linux kernel and making any changes they need. By the time the phone is released and anyone cares, it's an old version of the Linux kernel, and the changes weren't written in a way that it's easy to incorporate them in the first place. The idea of incorporating a large chunk of changes all at once at the end is nobody's idea of a good time.

I get that they have no responsibility vis a vis the GPL to upstream their changes, and I have no doubt that this is a cheaper way for them to run their business.

The SoC manufacturer provides to the OEM exactly one kernel version that works on that chip.

The OEM has to use that version. Oftentimes the SoC manufacturer will deliberately make it an old version because they know that the next android release will require a newer kernel version, therefore forcing the OEM to update to a new and more profitable SoC rather than continuing to sell an old model.

Isn't that a blatant GPL Violation?

Also; has any work been done on decompiling that kernel and trying to match up the symbols with existing source code?

The full source code of that kernel version is provided, including all KM drivers.

What I don't understand (due to my ignorance, please explain if you can)... what keeps the driver blobs from continuing to work from one version of Android to the next?

What prevents people from using the same drivers for a device on Android 7 and upgrading the OS to 9 or 10? I always understood that drivers would continue to allow the OS to recognize the component that the drivers are for unless something drastically changes on the OS side of things.

Why can't we say "Fuck you, Qualcomm" and continue to use the older drivers for newer OS versions without having to wait for them to officially support it?

This is because the Linux kernel itself is constructed and developed in a way where driver interfaces (API) are ruthlessly refactored and the linker interface for kernel modules (ABI) is intentionally version-incompatible. If the driver's source code is not in the kernel, it doesn't gain the benefit of kernel developers doing this ruthless refactoring and falls behind immediately from a source point of view, and from a binary point of view, modules compiled for one kernel intentionally will not work with another.

This is a conscious and opinionated strategy on the part of Linux, as an effort to make the cost of keeping drivers closed-source high and the cost of kernel refactoring low. It also has... disadvantages, in the form of a massive wasteland of only-supported-once snapshot-in-time devices. I am sure this comment thread will happy expound on this in great and painful detail.

Secondly, there was an entire effort by Google called "Project Treble" to build a stable Android HAL/ABI over the top of these unstable kernel interfaces. Unfortunately it would/will require essentially an Android-specific rewrite from vendors in order to fully implement, so many drivers are not yet operating in this model.

Project Treble is already fully supported by Qualcomm and other SoC vendors, that's not the issue.

The issue is that Treble simply shifts the compatibility issues from on from one layer (kernel) to another (Android HAL). There's still a compatibility horizon, because newer Android systems will eventually want to use newer kernel features and HAL APIs which were not present on ancient HALs which have been abandoned by the vendor.

And even if the OS could theoretically maintain backwards compatibility with older HAL APIs for 10 years, in practice the System + HAL combinations which aren't fully tested tend to break for various unforeseen reasons, making it unwise to push OS updates to devices which aren't fully supported by their vendor.

>as an effort to make the cost of keeping drivers closed-source high

I don't think it is intentional. Linux development is very pragmatic, they just want to have freedom to refactor anything without caring about stable kernel APIs.

Ironically, they care a lot about stability towards user space interfaces, to the point that bugs become features if someone depends on them, but they don't care about driver interfaces being stable.

This seems like a win-win. The kernel developers are free to refactor as they need to keep the quality and velocity high and companies are incentivized to upstream patches including making drivers in-kernel.

It's somewhat pragmatic for the Linux developers. Support a stable kernel API probably wouldn't be that expensive. It's clearly a political decision against writing close source code that does anything low level.

I am not sure why the comment is being downvoted - thats exactly what it is, a political decision to force companies to produce open drivers.

Because of this unsubstantiated and baseless claim:

> Support a stable kernel API probably wouldn't be that expensive

How is it clear?

> This is a conscious and opinionated strategy on the part of Linux

If anyone wants to read the Linux developers’ explanation directly, here it is:

https://www.kernel.org/doc/html/latest/process/stable-api-no...

> is a conscious and opinionated strategy on the part of Linux, as an effort to make the cost of keeping drivers closed-source high.

And Qualcomm is probably very, very happy with this arrangement. They can put a gazillion devs to maintain their BSPs using any means possible, but a small 2nd tier SoC fabbless don't.

Or you just push it upstream, which solves your problem because whoever is refactoring kernel code is also likely doing the appropriate changes in your code if necessarily.

This assumes that the person making the SoC owns all the rights to the code for their drivers, instead of licensing them from other firms.

You can demand those terms when you negotiate the license.

And they can say "no" or "money please."

Is it truly necessary that all these drivers run in the kernel space?

I bet a lot of drivers are just interfacing with a standard serial line which could be done in the stable user space and then exposing the next layer through something like FUSE.

If I understand treble correctly that’s close to what Android is doing, where each HAL can be done in userspace?

Didn't some Linux maintainers had very clear legal words for NVidia when they tried to make a kernel HAL/ABI?

NoyesNo, Nvidia tried to make a doggy legally questionable workaround for license aspects, so it's not quite the same.

Also as far as I remember (I might be wrong) the google HAL isn't a abstraction/redirection in the low level kernel module layer it lives on a higher abstraction level and was meant to make the amount of ways braking less but never planed to add abstractions on top of which you could build a graphics driver or CPU support (which doesn't only affect drivers but potential also kernel compiler time conditional compilations).

Still not as a bad as Linuxant: https://lwn.net/Articles/82305/

I'm having trouble understanding the issue was here. Linuxant produced a module that lied to the kernel about being a non-binary blob. Am I correct, but I'm still having trouble understand how the company survived since they seem to have a website up still.

Linux kernel license enforcement is very lax. Most binary blob drivers probably violate GPL in one way or the other, as do the "GPL-condom" half-open-source "drivers" that do most of their stuff in a userspace blob.

When asked about it most kernel developers point to closed-door negotiations that are being kept confidential. Usually also saying "trust us, we know what we are doing, we'll get more source opened our way".

Those who actually try to enforce GPL publically are ignored or shunned, e.g. Harald Welte/LaForge of gpl-violations.org

Are you implying the 'no stable binary ABI' policy is responsible for a lot of e-waste?

I mean, that seems to be the result, yes? The intent is to compel device manufacturers to release the source of their drivers, but at this point, I think we have to admit that's not a great description of what's happening in the mobile space.

> The intent is to compel device manufacturers to release the source of their drivers

I doubt that was the "intent"... perhaps a contributory argument in favour of a non-stable ABI, but I understand the reasons are far more pragmatic: maintaining a stable kernel ABI is a lot of work which could have strangled Linux in its early years.

> have strangled Linux in its early years.

And likely would still do so toady. I mean we are speaking about the kernel internal API (not the syscall API, which is stable).

Even systems which pride themself for internal kernel API stability do change the API every view years, while often having much less internal (feature/platform support) changes then Linux. Furthermore in android Linux LTS versions are often used which do support a long term stable ABI, potentially longer then certain other more niche systems priding themself with long support.

The problem is when a device is sold the kernel (minus back ported security fixes) is often already multiple years old so even if you use a kernel with 7 years longtime support you still have a good chance to run into problem. And supporting any kernel internal API for more then that time is unrealistic.

Why? WDDM drivers written for Windows Vista can still work on Windows 10. And Windows is not a "niche" system.

Because Windows is a (very broken) Micro-Kernel system which did very little changes to it's kernel (as far as I know) in the last many years, in difference to that Linux is a macro kernel with constant changes and improvements to it.

Are the improvements worth the cost? I don't know. (For server applications likely yes, but for desktop likely not).

What I do know is that there are clear benefits of micro kernel architectures, even if they are kinda messed up (for a micro kernel) hybrids like windows has.

Lastly the way the kernel API's are designed can also make a major difference independent of the rest. Honestly the Linux kernel API feels much more raw then the windows API, but I only have every written a single dummy driver for each platform and that was quite a while ago.

This comment feels like it was written by someone who hasn't done much Windows development (which is fine!). The kernel changes a lot over time, introducing entirely new driver models (see https://en.wikipedia.org/wiki/Windows_Display_Driver_Model#H... for an example), OS-wide features, etc. Things like thread scheduling or timer granularity can and will change (see https://randomascii.wordpress.com/2020/10/04/windows-timer-r... for a recent example)

Let's not forget about massive stuff like the whole Windows Subsystem For Linux, which was literally an entire separate process type alongside Win32 processes, with its own set of syscalls etc. Added and then deprecated (and probably eventually removed entirely) over the span of years.

Maintaining a stable kernel ABI is no doubt a big investment, but it's possible and doesn't have much to do with whether your kernel is a microkernel or not. Lots of Windows drivers these days are user-space which helps maintain compatibility and stability.

Windows NT is not a microkernel - Windows CE was, however.

NT has been shifting towards running more driver-code in user-space, as has macOS as well, especially with UMDF (especially for USB devices) and for graphics drivers, which are historically the #1 cause of BSODs - however just because the kernel passes control to user-space for the bulk of the driver's number-crunching doesn't mean that architecturally the kernel is still responsible for huge swathes of the computer's functionality. With a true micro-kernel there isn't any third-party code in the kernel space - not even a stub.

> however just because the kernel passes control to user-space for the bulk of the driver's number-crunching doesn't mean that architecturally the kernel is still responsible for huge swathes of the computer's functionality

I typo'd that - it should be

> however just because the kernel passes control to user-space for the bulk of the driver's number-crunching doesn't mean that architecturally the kernel isn't still responsible for huge swathes of the computer's functionality

Granted, even ABI-stability to some extend is difficult. I remember that I have some cheap wifi adapters that aren't supported by in-tree code, but came with a CD (small diameter) that had the SDK on it. This was unfortunately for the old timespec or w/e the exact struct is called, and I could have ported it to use the new API, especially considering that there are in-tree and/or up-to-date versions for closely related chips, but honestly the lack of easily-accessible tooling for that discouraged me from doing so.

The reason I got them initially is that, some about 2.5y back, they were sold on Aliexpress for 3.50 EUR (~5$) and could do dual-band with at least 40 MHz channels, iirc maybe even 80 MHz on 5GHz. They are just tiny USB-A dongles with a USB 2.0 HighSpeed host interface, but they could do station/softAP/station+softAP, obviously restricted to a single channel.

There's a related issue of a rtl8812-au or rtl8821-au (or compatible) stick (Edimax EW-7822UAC) I got before that, with a USB 3.0 SuperSpeed host interface, intended for trying out a remote radio head point-to-point configuration using polarization-duplex with a recycled satellite-TV dish on each end, as it could do AP-mode on 80 MHz wide channels in the 5 GHz band on Linux, with a 867 Mbit/s link rate (assuming full polarization multiplexing based on the stick's 2x2 MIMO and a circular polarizing feed antenna).

There are many different reasons that all contribute. But Linux clearly can keep some form of stability, given their dedication to never breaking userspace, if they really want to.

> The intent is to compel device manufacturers to release the source of their drivers, but at this point, I think we have to admit that's not a great description of what's happening in the mobile space.

But it's still the thing we want, for that and a lot of other reasons. So presumably what we need to do is find a way to exert more leverage, so that we actually get what we want, instead of folding and letting the bad thing take root forever.

I dunno, seems a stretch to blame the kernel for e-waste when theres tons of other well funded parties creating the waste in the first place with zero intention of supporting the device beyond a year, let alone enabling anyone else to support it.

The android ecosystem is already better than the completely proprietary feature phones of before.

No, Qualcomm is responsible for that. It's their device. The idea that a third party is responsible for supporting thousands of different devices simply doesn't make sense.

With new Android versions come new kernel versions, and the interfaces to drivers are not stable, and I assume Android-specific interfaces also do change.

Meanwhile, on a platform with a generally stable binary driver ABI alike Windows, even drivers for a prototype MSM8960 phone (Snapdragon S4) released in 2012-2013 will still run on 2021 versions of ARM32 Windows (insofar as these are still being built and released, of course).

Google is changing this for Android with GKIs (General Kernel Images) being mandatory[1] for new phones which launch with Android 12 (i.e. late 2022 phones). They'll maintain a stable ABI so that OEMs can plug in their proprietary drivers as kernel modules.

Additionally with the work the Android Kernel team are doing to upstream so Android Common Kernel can be just replaced with using the Linux mainline, we're getting ever closer to limiting the effect that closed-source drivers have on updates.

[1] https://source.android.com/devices/architecture/kernel/gener...

ABIs are only needed because some manufacturers insist on closed source blobs. Why should Linux, a project whose entire raison d'etre is to provide an open source kernel, make life easier for companies unwilling to accept that?

The questuon is, can you actyally force their hand? So far, the answer seems to be no.

Personally I feel we need to push for a law where the iser has a right to an open driver in the same way he has a right to a manual. Attempts to ship undocumented / secret functionality is software or devices should be equates with spreadind spyware or malware.

> A common offender is camera drivers

This is precisely why open source libcamera[1] exists.

[1] https://libcamera.org/

Android should have been GPL

And how would that make any difference when it comes to radio firmware?

The radio firmware could be executed on different CPU than the Android kernel and communicate with a standard protocol ?

That's more or less what happens already.

It is but if the radio driver was open source then the interface with the radio SoC would be documented.

Right now it is the worst of both worlds. Closed interface for a closed undocumented chip able to execute arbitrary code.

The interface isn't closed anymore, is it? [1]

[1] https://www.kernel.org/doc/html/latest/mhi/mhi.html

Drivers don't need to be open source to interact with GPLed code.

The part of Android that talks to the Qualcom drivers is the Linux kernel, which is GPL already.

You can read the patented algorithm in the patent, and even implement it, but you cannot legally distribute it, until the parent expires.

Apache License 2.0 is GPL-compatible, isn't it?

Edit: expanding the acronym, to evade Android/Apache confusion.

The whole point of the patent system is that you do not keep the technique obfuscated.

Yeah, I think this is a major design point behind Fuschia/Zircon; they have a stable driver ABI to address this specific problem