It's not really about the CPU architecture, but the peripherals of the devices. In the x86 world, the peripherals are governed by the "PC" standard, which these days is implemented by the chipset on your motherboard. They all have similar functionality, although you still do need some chipset-specific drivers in your kernel.
In the ARM world, the landscape is different. There has never been a de facto standard configuration of an ARM computer like the PC was/is for x86. There are ARM computers that are more like a micro controller and there are tablet/mobile devices based on ARM that have a system on chip (SoC) that's got all the peripherals, including GPUs, network adapters, etc. Every SoC is pretty different, there's a bunch of devices connected with i2c, spi and other buses. These are required to control the power and clock management and all the I/O devices.
ARM (the company) doesn't really design these, they just license ARM (the architecture) to a SoC manufacturer and call it a day. This leads to a very diverse and fragmented landscape.
The ARM world has been making strides to reduce the amount of chip-specific code. One good example is the device-tree configuration which, in theory, allows running the same kernel binary on different devices. With device-tree, you pass a config file from the bootloader to the kernel, which contains configuration (i/o memory maps, etc) of the peripherals (which you'd query from the hw/firmware in a x86 machine). This is a huge improvement over what was before it, but there's still a long way to go. And there's still plenty of older ARM SoCs out there that don't support this.
But yes, it's pretty much a mess. There's a lot of duplicate work that every ARM SoC manufacturer has to go through. Given that the mobile world is financed by a planned obsolescence cycle where they expect customers to upgrade every few years, I find it unlikely that this issue will be solved any time soon.
We had Z80, 6509, 68000 and so so, but everything else that was plugged into their pins was different.
So it doesn't matter if the Assembly is the same, the kernel also needs to talk to the hardware.
Also ARM just resells the base designs, each company usually adds they own set of Assembly instructions or fine tunes the designs in some other way.
The IBM PC clones were the only ones that had some kind of standardization going on, and even that seems to be gone now, with OEMs going to the same kind of hardware designs we used to have (everything bolted on the board).
https://wiki.debian.org/DebianKernel/ARMMP
Unfortunately most mobile devices are not supported by mainline Linux (N900 is best supported, but not fully), usually only server/desktop/laptop vendors have any incentive to mainline their drivers.
http://penguindreams.org/blog/android-fragmentation/
The TLDR is that it'd be nice if Android was more like Windows/Intel: install the OS (i.e. AOSP), drivers (or binary driver package format and an SDK to auto-build it) and boom you're done. Clean, stock, standard. Same with many Linux distros on x86/64.
I need to write an update to the article though. Lately I've been struggling with a Clearfog ARM board, and coupled with Torvald's recent comments on Intel vs ARM, I wonder if a huge chunk of the problem is ARM itself.
I've noticed for every distro there are really specific images, not just for each ARM chipset, but for individual boards, even when those manufacture patches make it into the mainline kernel.
I haven't looked closely at Android and I'm really curious at the build system/workflow used by things like Cyanogen and Omnirom. Are kernels really specialized per device? Does the ARM architecture itself make it difficult to have general purpose kernels that will boot on all of them like in the x86/64 world?
Any ARM hardware people care to chime in?