News for hackers. Stuff that matters.
Re:Android security in 2016 is a mess by djsumdog on Monday November 28, 2016 @02:09PM (#13057605)
One of the major major major issues is the hardware itself.

Let's jump back to the late 90s. You want to try Linux. You might partition your disk, or build a different machine and get a KVM or use an old laptop or add a second hard drive. In all cases, you can just install a Linux distro and see if it works. Maybe Ethernet or sound or something doesn't work. You could try a different distro with a newer kernel, maybe compile your own kernel with the modules you need, etc. But no matter what, it would at least boot (usually).

Intel x86/64 is a standard architecture. EFI makes the booting process even easier (although some motherboards still fuck this up; but for the most part fiddling with efibootmgr and turning off TPM will work. If not you can always turn on legacy BIOS and use the MBR). ARM is not a standard architecture. It's a spec that is sold to companies who all make their own SoC templates. There are things like Device Tree Config that make some of this more standard, but it's still far from x86/64. Just read Torvalds rants on ARM to learn more.

Even if you make some standard Android builds with all the drivers for all phones in a massive 20MB initrd, a lot of kernels are fundamentally different with the way bus connections are defined. Manufactures take forever to release their kernel code, often under threats by people for GPL violations, and even then they are filled with tons of binary blobs and shims that connect their .o files so kernel calls.

Stuff like Plasma looks amazing. It only has like two phones it supports. If you look at any android rom project, you see totally different builds for every single device. If Intel had been like this, Linux would have died off in the 90s, for both servers and desktops.

Device manufactures depend on you buying new phones. They cannot support phones after two years or else it kills their profit margins. Planned obsolescence like this leads to waste (don't kid yourself, e-waste recycling gets shipping to Africa where kids remove components. A small percentage of phones actually get recycled).

TL;DR Mobile phones are based on unmaintainable and non-standard hardware (intentionally). I wrote a post on this a while back:

http://penguindreams.org/blog/android-fragmentation/

Re:Android security in 2016 is a mess by 0xFFC on Monday November 28, 2016 @03:48PM (#13058278)
>Device manufactures depend on you buying new phones. They cannot support phones after two years or else it kills their profit margins.

This is the reason after using Nexus devices for 5 year, I will buy an iPhone and I will never ever look at Android again. My girlfriend switched from Galaxy note 4 to iPhone last year, and she is amazed how much iPhone experience is better than high-end Android experience.

2 year only update? Give me a fucking break, this total rip off. If I remember correctly even iPhone 5 does get update(and remember iPhone 5 is 5 year old phone, which will get new version of OS, not just security update, this alone is more than lifespan of Nexus or pixel device). Let alone the fact phone processors passed the threshold, phone produced after 2015 are almost as powerful as desktop from ~2005. My point is , if you don't play intensive games or any specific use case, an above ordinary phone would run browser,mail client, chat apps for more than 4 year without any issue. But companies trying to minimize lifespan just because of their own profit.

And my answer to them would be my middle finger.

They are totally screwing up customers.

Re:Android security in 2016 is a mess by deadringerr on Monday November 28, 2016 @08:21PM (#13059945)
I agree with you, phone companies have developed a planned obsolesce model that is very frustrating. I actually just went through the same process in the opposite direction - I switched from having iPhones to Android. I've heard tales of iPhones staying capable and functional for many years but my experience was that after a year or two there'd be an update that drastically reduced the performance (or rendered it basically unusable which happened 3 times). I suppose the grass is greener on the other side but I'm pleased so far with my switch. Good luck.
Re:Android security in 2016 is a mess by koffiezet on Thursday December 01, 2016 @01:04PM (#13081726)
In my experience with iPhones, this was true for older generations where they simply didn't have the processing power for newer OS features. My switch from my iPhone 4 to a 5 was strictly for performance reasons, but I went from my 5 to a 6S just to have something newer with mostly a better camera and touch-id. Performance-wise the phone was still perfectly usable (my brother still uses it). Sure it was a lot faster, and there still are massive leaps in performance with every generation, but these days that performance is mostly used to race to idle for improved battery life.

For the current applications, an iPhone 5S level of performance - a phone which is 4 years old - is still very usable today. That's also why Android flagship phones aren't really in trouble while still struggling to match the performance of a year-old iPhone 6S, and can't even dream of getting near the iPhone 7's performance. They mostly try to compete with stuff people do notice these days: the camera - where they have a lot more success.

Re:Android security in 2016 is a mess by qwertyuiop924 on Monday November 28, 2016 @07:41PM (#13059758)
...Cyanogen?
Re:Android security in 2016 is a mess by goalieca on Monday November 28, 2016 @09:46PM (#13060348)
For instance, if your kernel depends on Qualcomm snapdragon 800 support and Qualcomm no longer supports those closed bits for OpenGL and the rest, then even with cyanogenmod you'll never be fully patched. But cyanogenmod isn't the smoothest experience to be honest.
Re:Android security in 2016 is a mess by eon1 on Monday November 28, 2016 @04:48PM (#13058728)
"..remember iPhone 5 is 5 year old phone, which will get new version of OS, not just security update"

Yes, and remember that Apple intentionally writes OS updates that will run slower on their older devices, thus achieving almost the same thing but slightly less blatantly.

Re:Android security in 2016 is a mess by rrdharan on Monday November 28, 2016 @05:00PM (#13058799)
It maybe the case that OS updates run slower on older devices but I think it's a pretty strong statement to claim that this is "intentional", at least in the sense that you seem to be implying.

Do you have some strong evidence of this? Amusingly it's actually being tested in court (see the class action referred to in the link below):

http://www.cbronline.com/news/mobility/devices/apple-accused...

Re:Android security in 2016 is a mess by macintux on Monday November 28, 2016 @07:18PM (#13059600)
Some updates actually make older devices go faster, anecdotally.
Re:Android security in 2016 is a mess by yrro on Tuesday November 29, 2016 @08:39AM (#13062934)
My iPad 3 is still usable and receiving security updates. The idea of the same being true for an android tablet of the same vintage is hilarious.
Re:Android security in 2016 is a mess by pcwalton on Monday November 28, 2016 @02:22PM (#13057680)
I can't think of any reasonable definition of "standard" that admits x86 but not ARM. They're equally closed, proprietary ISAs.

It's the IBM PC architecture, not x86, that is the de facto standard. It's reasonable to prefer that to the relatively recent, and poorly adopted, device tree stuff that you see in the embedded ARM world. But let's not ding the ARM CPUs themselves for that.

Re:Android security in 2016 is a mess by oneplane on Monday November 28, 2016 @03:16PM (#13058036)
That's just the thing, there is no 'ARM CPU'. There is a ARM ISA, and some Core spec's, not a real 'ARM CPU'. Maybe a complete ARM SoC design you can buy, but nothing like an interchangable CPU, let alone socketed with multiple vendors.

If it's ARM-based, it's probably an embedded, closed, proprietary system with zero design need for 'compatibility'. With x86-based boards, you know there will at least be some sort of standard BIOS of EFI interface, a set of busses, a minimal number of devices that will behave and work the same on almost all systems. With ARM, there is no such thing, except for most parts of the ISA, and that's simply not enough.

Re:Android security in 2016 is a mess by kps on Monday November 28, 2016 @04:06PM (#13058415)

  > There is a ARM ISA
If only. It's possible to have two (32-bit) ARM processors with no instructions in common.
Re:Android security in 2016 is a mess by 3pt14159 on Monday November 28, 2016 @04:24PM (#13058558)
Wtf, really? As a machine learning developer with little interaction with low level stuff like mobile instruction sets, that sounds like pure madness.
Re:Android security in 2016 is a mess by wtallis on Tuesday November 29, 2016 @02:09AM (#13061407)
It's a bit misleading: he's referring to the Cortex-M series of ARM microcontroller cores, which do not implement the core ARM instruction set but instead only implement the Thumb series of alternate instruction sets that are optimized for code density. There are older ARM cores that pre-date Thumb and only have the core ARM instruction set, but there would never be a situation where you'd want to take binaries compiled for one of those devices and run them on one of the microcontroller cores.

There are however, plenty of other variables across the ARM ecosystem that can cause real headaches, like the different versions of the VFP floating point co-processor.

Re:Android security in 2016 is a mess by zeusk on Tuesday November 29, 2016 @02:38PM (#13065626)
Citation needed.

AFAIK, ARMv7 in user/system mode is backwards compatible with ARMv5 at-least. Even AARCH32 claims so.

Re:Android security in 2016 is a mess by kps on Tuesday November 29, 2016 @05:08PM (#13066663)
As wtallis said: Cortex-M executes only Thumb instructions, while pre-ARM7TDMI don't execute Thumb instructions.
Re:Android security in 2016 is a mess by zeusk on Tuesday November 29, 2016 @11:10PM (#13068429)
Cortex-A, Cortex-M, Cortex-R have different targets, ISA, System organization as far back as their origination. Not sure why you'd compare them, I was obviously talking about processors within the same segment, Cortex-R7 is backwards compatible with previous versions, Cortex-A72 is backwards compatible with previous versions, Cortex-M3 is backwards compatible with previous versions.
Re:Android security in 2016 is a mess by pcwalton on Monday November 28, 2016 @03:34PM (#13058166)
x86 chips are themselves almost complete SoCs at this point [1]. That's why you can buy things like Compute Sticks, after all.

It's true that Intel is by far the largest supplier of x86 chips, and so you get some level of "standardization" with things like Intel HD Audio. But that's more the effect of a monopoly than anything else. Competition breeds fragmentation, but given the choice I'd rather have competition.

[1]: https://www.techpowerup.com/img/15-08-18/77a.jpg

Re:Android security in 2016 is a mess by oneplane on Tuesday November 29, 2016 @03:55PM (#13066209)
While true, it wasn't always that way. The IBM PC with the BIOS was basicaly one of the first layers you'd have to be compatible with, and up to UEFI systems with CSM for BIOS emulation, it's been supported for about 30 years.

There was a time where the chipset did a lot more than it does now. Right now, it's the PCH or whatever it's called, and you even have those SoC's where you don't even need that anymore. You used to have the possibility of a standard BIOS with a chipset from someone else (SiS for example) and audio from someone else, as well as PCI controllers, PICs, RTCs, HBAs, UARTs, and CPU's. Or have an Intel chipset with a non-Intel CPU, also possible. You could even make a complete x86 system with no Intel chip inside (pun intended). Right now, that's something you probably can only do with an AMD system and that's about it.

Re:Android security in 2016 is a mess by niftich on Monday November 28, 2016 @04:35PM (#13058639)
Your points are valid but this clusterfuck doesn't excuse Android: both Microsoft and Apple were able to deliver working ARM-based systems not subject to these issues: Microsoft by mandating UEFI, Apple by reasons unknown to me, but presumably helped by the fact that they're a vertical.

As a few people have noted, it's not "Intel x86/64" that's standard, they just happen to be a dominant player right now. The original architecture is the "IBM PC", and its clones that are "IBM PC-compatible", that all descend from the same BIOS lineage.

IBM became dismayed how other companies were profiting from its architecture, and released the incompatible PS/2 architecture instead [1], which fizzled, and its good parts -- like 3.5" floppies, VGA, and PS/2 connectors -- were adopted by the clones. In the 1990s, Intel and Microsoft rose to prominence and began to exert more control and dictate the direction of the architecture with specs like 'PC System Design Guide' [2], and they and other vendors produced several improvements (ACPI, USB, PCIe, AHCI, SATA, etc.) which were adopted by broad community consensus, pushing the architecture forward.

ARM is different, because they make their money licensing their cores, so historically they had no interest in cultivating a general-purpose, "vendor consensus" platform. And Android's owner, Google, just wanted to get its services like Google Search, Google Maps, Google Mail, Google Now into as many people's hands and pockets as they could -- and they succeeded, despite the platform's haphazard nature.

Samsung knows that Google and the handset manufacturers are allies of convenience. Samsung is hard at work trying to cook up an OS and ecosystem for its hardware that's not contingent on Google's continual involvement, even allying with Microsoft to bring .NET to Tizen after its previous attempts saw no developer uptake.

Meanwhile, Google knows that continued press coverage of Android's security situation is hurting the platform. It's trying to re-frame the partnership it has with hardware makers, to design more hardware in-house, and to lean on its partners only for contract manufacturing -- a shift from the Nexus days, incompletely executed with the first Pixel phone, but likely more to follow. This will enable Google to control more of the software on the phone.

[1] http://www.mcamafia.de/pdf/pdfref.htm [2] http://tech-insider.org/windows/research/1997/0711.html

Re:Android security in 2016 is a mess by bitmapbrother on Monday November 28, 2016 @05:07PM (#13058842)
>Your points are valid but this clusterfuck doesn't excuse Android: both Microsoft and Apple were able to deliver working ARM-based systems not subject to these issues.

Then how do you explain all of the Windows Phones, from Microsoft and their OEM partners, that were stuck on the previous versions of the OS that are receiving no updates whatsoever?

>Samsung is hard at work trying to cook up an OS and ecosystem for its hardware that's not contingent on Google's continual involvement, even allying with Microsoft to bring .NET to Tizen after its previous attempts saw no developer uptake.

Let's see, if Microsoft wasn't able to make .NET work on their phones what makes you think Samsung has a chance with a platform that has even less marketshare and mindshare? Samsung is just doing what Samsung always does. Incidentally, Google also joined the .NET foundation for what it's worth.

Re:Android security in 2016 is a mess by ocdtrekkie on Monday November 28, 2016 @06:53PM (#13059459)
Windows 10 Mobile is not subject to these issues. Windows 10 Mobile has more restrictions on manufacturers than older versions, hence why they're "stuck behind".

Any OEM, for instance, that didn't give up control of software updates, couldn't get Windows 10. Even 8 had some more leeway, Dev Preview could install over the OEM build whether the OEM drivers were ready for it or not.

Re:Android security in 2016 is a mess by vezycash on Monday November 28, 2016 @08:34PM (#13059991)
Actually, there are Win 10 mobile devices stuck in some old build.

Also, there are also some devices from some OEMs that are not supported in the insider program.

Re:Android security in 2016 is a mess by ocdtrekkie on Monday November 28, 2016 @10:37PM (#13060577)
Those are the cases of devices never supported with Windows 10 in the first place. Dev Preview still worked (for a while), but it never officially got 10.
Re:Android security in 2016 is a mess by remir on Monday November 28, 2016 @06:30PM (#13059339)
Google is also developing a new OS, Fuchsia, with a new micro-kernel (Magenta). It uses Dart/Flutter/Meterial Design for apps development.
Re:Android security in 2016 is a mess by goalieca on Monday November 28, 2016 @09:42PM (#13060333)
Any news as to whether this is a real android replacement or just a research OS like plan9 and singularity??
Re:Android security in 2016 is a mess by pjmlp on Tuesday November 29, 2016 @02:36AM (#13061489)
Given the state of the Dart/Flutter demo app, I imagine they still have a long way to go before anything gets out the door, if ever.

https://play.google.com/store/apps/details?id=io.flutter.gal...

Re:Android security in 2016 is a mess by remir on Tuesday November 29, 2016 @05:34PM (#13066806)
Not much is known at the moment, but it is developed in the open. https://fuchsia-review.googlesource.com
Re:Android security in 2016 is a mess by ocdtrekkie on Monday November 28, 2016 @02:32PM (#13057759)
How has Windows Mobile solved this? Dev preview is available for all phones, updates are pushed immediately from Microsoft regardless of the manufacturer. Firmware with drivers are released separately from the OS. All on the same ARM architecture.
Re:Android security in 2016 is a mess by golf1052 on Monday November 28, 2016 @03:00PM (#13057915)
Maybe because they require hardware manufacturers to use a known set of processors (see https://msdn.microsoft.com/en-us/library/windows/hardware/dn...)
Re:Android security in 2016 is a mess by my123 on Monday November 28, 2016 @03:03PM (#13057935)
Getting Windows go boot on the completely fictuous qemu -M virt platform for both arm and arm64 was very easy, USB and SCSI worked after less than a week of fixing Qemu emulation bugs (GIC).
Re:Android security in 2016 is a mess by oneplane on Monday November 28, 2016 @03:18PM (#13058049)
You're basically patching the hardware to work with immutable software (as you won't have the windows sources). At the same time, the QEMU ARM virtual system and the Microsoft designated ARM SoCs are perhaps closely compatible whereas it would be very different with other ARM implementations.
Re:Android security in 2016 is a mess by my123 on Monday November 28, 2016 @03:28PM (#13058117)
Only GIC (interrupt controller) bugs needed to be fixed. The GIC is standard since the Cortex A9. On the A9, you have to use an HAL extension (declared in ACPI) for the timer(a sample is available on MS' Git repo). On the A7/A15 and later, Windows uses the GTDT ACPI table only, so nothing is needed :)
Re:Android security in 2016 is a mess by my123 on Monday November 28, 2016 @02:55PM (#13057882)
Windows on ARM uses UEFI and ACPI, it can even boot with zero drivers on Cortex A15-class(architectural timer) and later HW. Of course, you only have USB and eMMC in such a config, the driver model is also stable.
Re:Android security in 2016 is a mess by rodgerd on Monday November 28, 2016 @05:26PM (#13058968)
Microsoft care. Google don't.
Re:Android security in 2016 is a mess by jupiter2 on Tuesday November 29, 2016 @02:48PM (#13065706)
Late response: I'm having a hard time understanding your reasoning here.

A less generic comparison is Apple. Apple builds it's own devices and supports them.

If I buy an ASUS Tablet, I'm not expecting them to provide security updates to Lenova or Samsung Tablets. I am expecting them to support their own devices. They have everything they need - they built the devices themselves.

Similarly, if I buy an phone that's been purposely tweaked by AT&T, working with Samsung - I expect that they are more than able to provide security updates but are simply unwilling to do so. They have all the tools (drivers, access, etc) they need.

You're looking at the general Android landscape. It's a question of holding the device makers accountable. Providing security updates for devices you build yourself, with drivers you developed yourself (or through a partner) isn't an issue - it's pure (litigable) greed.

Re:Android security in 2016 is a mess by zzzcpan on Monday November 28, 2016 @02:31PM (#13057758)
Or maybe the problem is in Linux itself not being well suited to run manufacturers' blobs independently from the rest of the kernel?
Re:Android security in 2016 is a mess by subway on Monday November 28, 2016 @02:36PM (#13057782)
It's far worse than that. If it were simply a matter of running mfg's blob's, this would be an easy problem. Instead manufacturers fork the kernel, then patch the hell out of it (often in horrific-never-going-to-be-accepted-upstream ways).
Re:Android security in 2016 is a mess by euyyn on Monday November 28, 2016 @03:32PM (#13058147)
I'm not familiar with any of the specific reasons why manufacturers fork the kernel. But an obvious question to be addressed is: Can the kernel be modified, so they can achieve their use cases without needing to fork it? By making it more modular in some parts, or adding hooks they can attach to...
Re:Android security in 2016 is a mess by 91bananas on Monday November 28, 2016 @03:36PM (#13058187)
Same wonder here, what are the reasons they do this?
Re:Android security in 2016 is a mess by TD-Linux on Monday November 28, 2016 @04:04PM (#13058396)
It's quicker and easier than submitting any patches upstream. They don't care about support past product release.

Usually the SoC vendor provides the forked version, and the manufacturer forks that again, so they are twice removed from upstream. If they ever upgrade Android versions, they port all of their patches onto a new fork from the SoC vendor.

Re:Android security in 2016 is a mess by euyyn on Monday November 28, 2016 @05:52PM (#13059139)
The question is not "why don't they upstream the changes". Rather why do they have to fork to achieve what they need.
Re:Android security in 2016 is a mess by kbenson on Monday November 28, 2016 @06:34PM (#13059358)
Because kernel developers can only work with what they're given. If the hardware isn't supplied by the company or some grant, or if it's IP encumbered, then you have a situation where it's unlikely to be supported natively by the kernel.

At that point, it becomes a short-term or long-term benefit assessment. Long-term, it's better to work to get it accepted upstream, but that's more work in that you have to make sure your patch is acceptable and conforms to how the kernel devs want it, and they can verify the support works, instead of just working as you've set up in-house. With the rush to market, and with the specific configuration of chips possibly not being used in the next item released, short-term patches likely look really attractive.

Re:Android security in 2016 is a mess by euyyn on Monday November 28, 2016 @07:33PM (#13059697)
I don't think I'm explaining myself well. Can't the kernel be modified so that manufacturer-specific code can stay in-house?

Let me give you an example: I don't need to fork my text editor to make it highlight and autocomplete any new programming language I invent myself. And I don't need to give the editor developers anything to work with either. Because they've put the needed hooks and interfaces in place for me to write my stuff without having to modify theirs.

Re:Android security in 2016 is a mess by rogerbinns on Monday November 28, 2016 @08:19PM (#13059936)
> I don't think I'm explaining myself well. Can't the kernel be modified so that manufacturer-specific code can stay in-house?

The kernel is under the GPLv2 so redistributing kernel binaries also requires making source available (that is all part of the "freedom").

The kernel internals constantly change. For example the USB stack was rewritten 4 times. Each time all the drivers etc were converted over. By contrast Microsoft has also rewritten their stack 4 times, but has to retain backwards compatibility for each generation of driver API/ABI, simultaneously!

While in theory it is possible to have a stable API, in practise real life is more complicated. There are concurrency issues, power management, suspend/resume, memory allocation, constantly changing hardware etc. A stable API would result in way less optimal drivers (eg they may consume more memory or power, cause cores to run at higher speeds).

The kernel developers decided they weren't going to have that layer of indirection. Instead drivers will do the one true correct thing (updated as kernel versions change). That makes them simpler, and perform better in the dimensions mentioned in the previous paragraph.

That all said, Linux is fanatical about the userspace API to the kernel being stable. You can implement drivers (eg USB) and filesystems (eg FUSE) amongst other things in userspace. However they won't be as "good" as kernel drivers.

Here is a good post from a kernel developer: http://www.kroah.com/log/linux/stable_api_nonsense.html

To get an idea of what changes in the kernel, check out the LWN page. If you have interest in Linux, I'd recommend subscribing too. https://lwn.net/Kernel/

Re:Android security in 2016 is a mess by euyyn on Monday November 28, 2016 @09:46PM (#13060347)
Thank you for the explanation and the link to that article. I disagree with some of the philosophical statements, like:

> While in theory it is possible to have a stable API, in practice real life is more complicated.

(In practice it's also possible, as Microsoft demonstrated with Windows for decades).

or (from the article):

> benefits if your driver is in the main kernel tree, all of which has made Linux into such a strong, stable, and mature operating system which is the reason you are using it in the first place

(The reality is, that wasn't enough of a reason to move users from Windows, before the smartphone era. Nor is it the reason Linux won over Windows in the smartphone market, either.)

I think the article hits the crux of the issue, though: Nobody wants to spend their free time contributing to maintain an old interface, instead of working on the new one. A side effect of this policy is a better kernel, granted (for the devices that actually make it to the main tree).

So given that a stable API isn't an option, and the current model is clearly not working, I guess the solution lies in solving the hurdles these companies face to upstream their code. Do you know about that, or where could I learn more?

Re:Android security in 2016 is a mess by rogerbinns on Monday November 28, 2016 @10:38PM (#13060580)
Microsoft only had to support one architecture, and continuously changed driver models. Except 32 and 64 bit. Plus you can't use drivers across Windows versions that much. But at the end of the day it was a financial decision by them to maintain a very large degree of backwards compatibility, and by their customers to pay that price. But also remember just how "sticky" Windows 98, XP etc were. People wouldn't give them up. Also have you tried to use USB 3 on Windows 7? Or how about skylake until the backpedal?

Note that with Linux you can get more a semblance of Microsoft style stability. Use RHEL (redhat enterprise linux) and they do the work of backporting relevant drivers and changes. You also get to pay handsomely for that.

Note that the majority of devices do make it to the kernel. You can plug in virtually anything and it works!

You really should read LWN. They do a fantastic job of covering various issues. For example here is an article on hurdles: https://lwn.net/Articles/647524/

Re:Android security in 2016 is a mess by euyyn on Monday November 28, 2016 @11:41PM (#13060841)
With this:

> Note that with Linux you can get more a semblance of Microsoft style stability. Use RHEL (redhat enterprise linux) and they do the work of backporting relevant drivers and changes. You also get to pay handsomely for that.

> Note that the majority of devices do make it to the kernel. You can plug in virtually anything and it works!

it looks like you're mostly thinking of the server space. The problem of manufacturers forking the kernel, and the topic of this HN post in general, is on Android. E.g. my last two phones, a Samsung Galaxy Nexus, and an LG Nexus 5, won't get any more OS updates.

I'm going to read that article, and check LWN regularly.

Re:Android security in 2016 is a mess by rogerbinns on Tuesday November 29, 2016 @10:27AM (#13063582)
No, the majority of devices do really work. Plug in a random webcam, drive controller, spi peripheral etc.

Embedded systems are different, but it isn't the device driver support that is the problem. x86 has a mostly standardised architecture, and due to an accident of history has a I/O address space that is separate from memory address space. Consequently it has been possible to probe for devices with very low probability of collateral damage. Which is why Linux and Windows can "just work" on virtually any x86 system.

In the ARM world, you get the CPU design itself from ARM and copy/paste that into your design program. Then you copy/paste in other pieces you want such as memory controllers, USB controllers, serial ports, displays, storage, SPI, and whatever else fits your needs. Then you hit print and get chips with all that working together. There is no separate I/O address space, so these devices all end up in memory address space. There is no standard location for them. Somehow you have to know the USB controller is at address 0x12345678, and you can't practically scan the address space trying to find all possible devices, and certainly not without collateral damage.

So what would happen is system developers would fork the current kernel, and make a permutation of an existing system but with hardcoded knowledge of where all the pieces making up the system live in the address space, and how they are connected. Then they'd add support for unique devices, connections, quirks and bugs. Throw in some binary blobs, NDAs, "IP" etc and they now have their own unique kernel. And that is why they get stuck on a kernel version that can't be practically updated.

The kernel developers adopted device trees as a solution. This is provides a data description to the kernel at boot time of what platform devices are present, how they are connected and where to find them. That allows the same kernel binary to support a very wide range of hardware. ARM64 systems also typically require ACPI, which if you squint allows the platform to also provide code that can run for dealing with the platform.

What this means is that the problem has been addressed technically. It does not address financial issues: the vendors get no more money after first sale, so they don't care about updates. It also does not address the folks doing binary blobs, NDAs and other "IP" concerns. The Linux (GPL) attitude is all about user freedom (as in freedom of speech), and gives everyone participating an equal playing field (no party has a special position).

Re:Android security in 2016 is a mess by euyyn on Tuesday November 29, 2016 @03:43PM (#13066094)
Hey, I wanted to thank you for taking the time to explain this. The situation and its context is much more clear to me now.
Re:Android security in 2016 is a mess by rogerbinns on Tuesday November 29, 2016 @11:07PM (#13068414)
You are welcome. It is actually something that has been very painful for me and the project I've been working on. The newest kernel we had for ARM boards we were considering was 3 years old, and hence missing a lot of newer device drivers and other functionality, not to mention missing bug & security fixes.
Re:Android security in 2016 is a mess by voltagex_ on Monday November 28, 2016 @10:39PM (#13060587)
Two groups probably have more experience in that than others:

Linaro: https://www.linaro.org/

OpenWRT: https://www.openwrt.org/

Going on a bit of a tangent:

Most of the time these companies just don't care.

If you don't mind getting your hands dirty, then have a look at some vendor kernel trees. For example, a $300AUD router released last year is running 2.6.34brcm - http://www.tp-link.com.au/download/Archer-D9.html#GPL-Code

The 802.11AC card in that router will likely never have open source drivers. I'm pretty sure the routing engine is closed source too, although with 2x Cortex A9 chips I would have thought you could get away with doing it in software.

Re:Android security in 2016 is a mess by wtallis on Tuesday November 29, 2016 @02:30AM (#13061474)
> (In practice it's also possible, as Microsoft demonstrated with Windows for decades).

There may be a handful of narrow cases where Microsoft has preserved third-party driver compatibility across long spans of time, but when looking across a decade or more, they break as much as they preserve. Consider DirectX, which is now completely antithetical to its original purpose and name of allowing applications (games) relatively direct hardware access. Modern systems can't survive without a level of hardware abstraction and sharing that mid-90s systems couldn't afford. The end result is that you get essentially no hardware accelerated graphics on decade-old GPUs anymore, except where the drivers were re-written for recent versions of Windows. Vista's new audio subsystem killed off an entire product segment of hardware-accelerated audio processing.

Re:Android security in 2016 is a mess by euyyn on Tuesday November 29, 2016 @03:48PM (#13066143)
Oh, I meant the demonstration has lasted decades, not necessarily that the backwards-compatibility was eternal. Once you take the stance of preserving compatibility, there's a strong market force to deprecate and eliminate old interfaces and guarantees once it's practical. On both the PC market of then, and the smartphone market of today, hardware gets replaced faster than once per decade.
Re:Android security in 2016 is a mess by pjmlp on Tuesday November 29, 2016 @02:39AM (#13061497)
Somehow the majority of other operating systems manage to have such stable APIs.
Re:Android security in 2016 is a mess by zanny on Monday November 28, 2016 @10:34PM (#13060566)
The shit disasters vendors force into their forks would never be accepted mainline, and even if they would be, these vendors take the route of most control - working with anyone else would violate their own ego that they always know best, so they never work with anyone else.
Re:Android security in 2016 is a mess by subway on Monday November 28, 2016 @04:06PM (#13058416)
I can't speak definitively for any specific project, but I gather much of this is due situations like this:

SoC Mfg designs a new chip, heavily leveraging IP from an old chip. Since some old IP is still around, and already (mostly) has kernel support SoC Mfg, patches the existing code to work on the new SoC, publishes the bin/source as an SDK, and declares their work done. (The linux-meson/sunxi geeks will eventually clean this up and mainline it anyway).

In the mean time, Board Mfg needs to get a product out the door, so they start with the SoC and SDK, and find that a bunch of the work done in the SDK is really specific to SoC Mfg's reference board, and requires a few tweaks, so they hack on the SoC Mfg kernel tree until the device mostly works, declare a success and publish a board SDK. (Those same geeks from above will handle the mainline work if they want to run a mainline kernel anyway, so why should Board Mfg bother?)

In some cases, this cycle gets to repeat yet again when the board gets OEM'd into a half dozen devices, each requiring slightly different kernel tweaks.

Re:Android security in 2016 is a mess by zzzcpan on Monday November 28, 2016 @04:23PM (#13058554)
I think with microkernel architecture this could have been very different. Even touching the kernel would keep outdated only a tiny part of the system with a relatively small attack surface, the rest could still get updates.
Re:Android security in 2016 is a mess by ethbro on Monday November 28, 2016 @05:57PM (#13059170)
It would have, but in 1991 it would have taken some powerful crystal balls to architect towards a fragmented mobile hardware ecosystem unified by a JVM-run app platform.
Re:Android security in 2016 is a mess by rogerbinns on Tuesday November 29, 2016 @01:19PM (#13064958)
Microkernels wouldn't really help. The point of a microkernel is to drastically reduce the coupling between parts of a system. Current hardware has far more coupling between parts. For example power management requires coupling (consider a NIC on a USB controller on a PCIe bus) where all pieces need to pull together. Or consider something like scheduling in a world where processor cores aren't identical and what runs where matters.

Your microkernel would either perform way worse (eg worse power consumption), or have pieces so tightly coupled that you'll wonder if it was worth it.

Re:Android security in 2016 is a mess by lake99 on Monday November 28, 2016 @05:27PM (#13058977)
Well, they will still have to import all the upstream changes, and make sure that all the parts play well with each other, which, they can still do. Except that they don't.

I think what makes it hard is the dizzying variety of CPUs that makes it all hard to keep up with. I have heard from insiders that as far as manufacturers are concerned, a CPU stays "current" for about 6 months.

Re:Android security in 2016 is a mess by mohaine on Monday November 28, 2016 @04:19PM (#13058522)
They don't usually "Fork" much of kernel, they (mostly) just have drivers(modules) outside it. When refactors happen to kernel internals, they don't happen to external modules so it is hard to get them up to date with the latest version. Over time this gets worse and worse.

That said there are often some small changes to the kernel itself, which are also an issue. This is more common if the hardware implements something new that hasn't really been standardized in the kernel yet.

Re:Android security in 2016 is a mess by euyyn on Monday November 28, 2016 @05:53PM (#13059149)
Oh, so it's mainly a matter of the kernel not offering a stable ABI for drivers?
Re:Android security in 2016 is a mess by rodgerd on Monday November 28, 2016 @05:24PM (#13058955)
LWN reported on a survey: https://lwn.net/Articles/647524/

Of course the kernel community looked at the "Too hard" and "afraid of upstream" responses and called the programmers lying lazy jerks, which I guess proves the point.

Re:Android security in 2016 is a mess by avar on Monday November 28, 2016 @06:08PM (#13059236)

    > Let's jump back to the late 90s. You want to
    > try Linux [...] get a KVM [...].
KVM as in Kernel-based Virtual Machine? Don't you have the late 90s confused with the late 00s?

The first Linux release with KVM was in 2007, KVM came out of an Israeli startup around 2005.

Re:Android security in 2016 is a mess by barik on Monday November 28, 2016 @06:10PM (#13059248)
Re:Android security in 2016 is a mess by avar on Monday November 28, 2016 @06:43PM (#13059400)
I stand corrected and significantly less confused, thanks.
Re:Android security in 2016 is a mess by mtgx on Monday November 28, 2016 @02:19PM (#13057666)
> If Intel had been like this, Linux would have died off in the 90s, for both servers and desktops.

That's the thing. You said "Intel". Intel is not an architecture. It's a company - that happens to dominate the x86 market. Do you think the situation would be the same if there were 10 different x86 chip companies with at least 5% market share each? If we wanted everything to just work, as it does on "Intel", then we should probably wish for a Qualcomm monopoly with 90% of the market. And hope it's profitable enough that they can support their own chips for more than 18 months.

I can only describe Intel's chips as a rip-off for the most part (and it's only gotten worse in recent years). I mean, they're now selling Atom chips for $160 (about 4-5x more than a similar performance ARM chip) to notebook manufacturers, with very little difference in performance between them and the older Atom for smartphones. So I would certainly hope Intel's chips would be better supported. Although I actually don't think its Atom chips are as well supported as its Core chips.

Now, not having a single company completely dominate the ARM ecosystem is certainly a "problem" from this point of view. However, I still mostly blame Google, who mainly just wants its ads to run well on these devices, which seem to do that rather well in the current situation, as well as OEMs who want to be "different" and are probably even hostile to the idea of allowing say an LG ROM run on a Samsung phone. I would try to fix this Google/OEM situation first and aim my complaints at them.

Re:Android security in 2016 is a mess by userbinator on Monday November 28, 2016 @10:21PM (#13060508)
Do you think the situation would be the same if there were 10 different x86 chip companies with at least 5% market share each?

In the 90s, that's actually what happened.

https://en.wikipedia.org/wiki/List_of_x86_manufacturers

Re:Android security in 2016 is a mess by wtallis on Tuesday November 29, 2016 @02:44AM (#13061522)
Tracing the history of those various competitors is really interesting. Some (Transmeta) failed completely in a relatively straightforward manner and just disappeared. At the opposite end of the spectrum, NexGen became AMD's future when AMD's in-house design was a flop. Parts of Cyrix ended up with VIA and parts with AMD. Almost all of the big semiconductor companies either owned one of these x86 design firms for a time, or sold those chips under license.
Re:Android security in 2016 is a mess by AlphaSite on Monday November 28, 2016 @02:54PM (#13057880)
Well, there are 3.
Re:Android security in 2016 is a mess by djsumdog on Monday November 28, 2016 @07:32PM (#13059690)
Intel, AMD and... VIA?
Re:Android security in 2016 is a mess by nickpsecurity on Monday November 28, 2016 @11:11PM (#13060713)
VIA, thanks to Centaur, was first with the low power plus had PadLock Engine with TRNG and crypto acceleration. Their ARTIGO's made nice little boxes for $300.