News for hackers. Stuff that matters.
Re:How the ARM32 Linux kernel decompresses by jandrese on Thursday August 13, 2020 @02:25PM (#24146504)
I have to wonder just how relevant all of this effort is anymore. They're talking about faster reads and saving space, but the savings is seven megabytes. And even then it comes at the cost of leaving dead space underneath your kernel space in main memory where the old compressed kernel used to be.

They're doing all of this on systems frequently running off of 16 or 32GB flash devices. The difference in cost between manufacturing a 16MB flash device and a 512MB flash device these days must be minuscule.

Re:How the ARM32 Linux kernel decompresses by leeter on Thursday August 13, 2020 @02:33PM (#24146604)
It's cents... but consider this. Linux is run on devices embedded in just about everything. Places where even doubling from 128 to 256MB of flash ROM could seriously hurt margins. So as the old saying goes.. a penny here and penny there sooner or later you're talking real money. And it is real money sometimes millions of dollars in margin.
Re:How the ARM32 Linux kernel decompresses by andyjpb on Thursday August 13, 2020 @03:29PM (#24147379)
Cents is a lot of money in manufacturing.

When I made consumer electronics our production team would swap parts based on cost. 1c on a million units is $10,000. They only had to save a few cents a year each to easily pay for themselves.

The rest was margin for the business.

Saving cents on a $80 BOM (bill of materials) may not sound like much but it really does add up when you're doing volume manufacturing.

Re:How the ARM32 Linux kernel decompresses by andyjpb on Thursday August 13, 2020 @03:32PM (#24147416)
On the other hand, manufacturing has the general attitude that "software is free" because the incremental cost in production to make a copy is zero.

Once, the production team swapped a component as part of a running change and it took a programmer nearly a month to write the new driver.

I think we still came out ahead in the end but there was some worry along the way as it wasn't clear how long it was going to take.

Re:How the ARM32 Linux kernel decompresses by leeter on Thursday August 13, 2020 @03:38PM (#24147518)
Exactly this, you can even look at the old consoles and see the same thing. They were built to a VERY specific margin and tuned for the same. There is as best I can tell no fat in any of the systems shipped except newer consoles and the original xbox. The only reason I'd say there is 'fat' in newer consoles is because they have much wider requirements than old ones do.
Re:How the ARM32 Linux kernel decompresses by Dylan16807 on Thursday August 13, 2020 @07:46PM (#24151327)
And if I learned anything from the recent fiasco over BIOS support for AMD's Zen 3 chips, it's that 128 megabit flash ROMs have massive availability and widespread use. That specific size is very cheap, and going above it is disproportionately expensive.

And at that size, compression more than doubles what you can pack alongside the kernel.

Re:How the ARM32 Linux kernel decompresses by fest on Thursday August 13, 2020 @02:54PM (#24146873)
For NAND flash, maybe the difference is miniscule. For NOR flash, which is what many small embedded devices run from, the difference is huge: 128Mbit (16MiB): 1.01$ 2Gbit (256MiB): 16.38$

2Gbit is the highest commercially available package (with very few suppliers, so it carries extra risk).

Re:How the ARM32 Linux kernel decompresses by segfaultbuserr on Thursday August 13, 2020 @06:08PM (#24150109)
See also: The most expensive chip in the ARM world by Marcin Juszkiewicz [0]. It's written by a core AArch64 developer on the lack of a full-function boot ROM on most ARM single-board computers, forcing developers and users to fight with vendor-specific bootloaders.

> So SBC… Which means playing with U-Boot. In the past it meant random fork of it. [...] you have bootloader(s) built. The next step is taking microsd card, finding out which offsets to use. Then put bootloader(s) at proper places so CPU can read them and boot. Usually it also means that you have to use MBR style formatting [...]

The reason? A EEPROM chip is too expensive, an astronomical cost of $3! The author was half-joking, but only serious - $3 is really expensive for most single-board computers.

> The solution is SPI flash chip. But it is very expensive - 3 (three) EUR for 512Mb (64MB) chip (if bought 1000 of them). Far too much for 99.99% of SBC probably. Probably 64/128Mb will be enough even. And they are cheaper (128Mb is 1.7 EUR). Still too expensive…

[0] https://marcin.juszkiewicz.com.pl/2020/01/29/the-most-expens...

Re:How the ARM32 Linux kernel decompresses by pantalaimon on Thursday August 13, 2020 @03:02PM (#24146980)
Are people running Linux of NOR flash? On MCUs you usually have other systems like Zephyr, MBedOS or RIOT.
Re:How the ARM32 Linux kernel decompresses by fest on Thursday August 13, 2020 @03:10PM (#24147080)
Pretty much every security camera, most of the networking equipment, action cameras and set-top boxes run Linux from NOR flash (cost and simplicity reasons- no FTL, IIRC better retention). Altogether it's probably billions of devices.

There are also quite a few lines of MCUs without built-in flash memory and those typically boot from external NOR flash.

Re:How the ARM32 Linux kernel decompresses by segfaultbuserr on Thursday August 13, 2020 @05:49PM (#24149844)
NAND is the new industry standard, but most embedded systems, several billion units, produced in the early 2010s were powered by NOR flash, and many use Linux. One example was the vast majority of home routers. I once replaced the 4 MiB NOR flash in my home router to 16 MiB for OpenWrt (yes, you can run Linux with 4 MiB of NOR flash, this configuration was officially supported by OpenWrt until it's retired in 2019).
Re:How the ARM32 Linux kernel decompresses by dmitrygr on Thursday August 13, 2020 @02:34PM (#24146619)
The bootloader's driver for your storage device is often very simplistic. Eg: it might only read one sector at a time, might not have the FAT cached in RAM, etc. It is not designed for speed - only for simplicity and size. So, reading a few MB fewer from storage is often actual time savings.

Plus there are things like [e]MMC's "boot" area, where a special storage area is simpler to read using a simpler special protocol. That area has a limited size.

Re:How the ARM32 Linux kernel decompresses by jandrese on Thursday August 13, 2020 @02:43PM (#24146734)
What kind of time savings are we talking about for 7 megabytes here? A second? Three seconds? Or is it more like 20ms? Is it still common to source components from 1995 in these devices?
Re:How the ARM32 Linux kernel decompresses by setpatchaddress on Thursday August 13, 2020 @02:55PM (#24146875)
20ms is a long time in low-level systems programming, especially where concurrency might be low or limited. You only need to find five 20ms savings opportunities to shave a tenth of a second off of your boot time.

Edited to add: A lot of performance loss for large, complex systems is death-by-a-thousand-cuts -- 20ms here and there can make a real, user-perceivable difference.

Re:How the ARM32 Linux kernel decompresses by tkln on Thursday August 13, 2020 @04:33PM (#24148567)
I don't have exact numbers on this any more. It's been couple of years since I last worked in embedded storage and boot optimization.

When optimizing boot time of an embedded system something like 100 ms is a significant saving once you've picked all the low hanging fruit in the user space init side. In the end, every millisecond counts as it all adds up when there are hundred or so places where they may end up being spent needlesly. An embedded product, especially automotive, might have rather strict boot time requirements.

The time it takes for a bootloader to load a kernel image can be a decent part of the boot time. The bootloader most likely won't be running the SDHCI controller in any of the faster modes that require implementing bus training, calibration, and tuning algorithms of varying complexity.

Re:How the ARM32 Linux kernel decompresses by MartijnBraam on Thursday August 13, 2020 @04:08PM (#24148046)
In case of the PinePhone (which is arm64) the actual loading of the kernel image from eMMC can take a second because the MMC controller isn't initialized for max speed by the bootloader, so having a smaller kernel saves some time between pressing the power button and having the splash screen show up (since the bootloader can't control the display)
Re:How the ARM32 Linux kernel decompresses by megous on Thursday August 13, 2020 @05:22PM (#24149433)
U-Boot can absolutely use the max speed. Just apply some mmc patches from here: https://megous.com/git/u-boot/log/?h=opi-v2020.04
Re:How the ARM32 Linux kernel decompresses by voltagex_ on Thursday August 13, 2020 @07:15PM (#24150896)
Do you think those patches will be upstreamed?
Re:How the ARM32 Linux kernel decompresses by megous on Wednesday August 19, 2020 @07:09AM (#24209135)
If someone does it, then yes, otherwise I don't plan to do it myself.
Re:How the ARM32 Linux kernel decompresses by voltagex_ on Thursday August 13, 2020 @07:14PM (#24150880)
Slightly off topic: the last time I saw stuff about speeding up boot of small devices was here - https://megous.com/git/p-boot/about/
Re:How the ARM32 Linux kernel decompresses by megous on Thursday August 13, 2020 @05:24PM (#24149472)
Usual unoptimized speed may be ~23MiB/s for SD card for example.
Re:How the ARM32 Linux kernel decompresses by pm215 on Thursday August 13, 2020 @02:53PM (#24146857)
It might (or might not) be less relevant, but it's also to a great extent already-expended effort. Arm kernels have had a built-in decompressor for over a decade, so the bulk of the effort went in many years back when the gains were clear and worthwhile, and now while there's some maintenance cost to keeping it rather than just ripping it out, it's not a big cost. And dropping the decompression has the catastrophically huge cost of breaking an enormous number of already-deployed systems when they update to a decompressor-less kernel.

Incidentally, aarch64 kernels do not include a decompressor -- if you want to compress the kernel image then it is the job of the bootloader to uncompress it. Different primary market, different time, different constraints, opportunity to choose differently, different outcome...

Re:How the ARM32 Linux kernel decompresses by gspr on Thursday August 13, 2020 @02:47PM (#24146780)
I mean just today I had to contemplate whether I could afford including a printf implementation in my code. Granted, this thing does not run Linux, but there's plenty of resource-constrained systems out there that do.
Re:How the ARM32 Linux kernel decompresses by secondcoming on Thursday August 13, 2020 @02:34PM (#24146624)
It's miniscule for one, but grows quite large when you're intending to make millions of something.
Re:How the ARM32 Linux kernel decompresses by segfaultbuserr on Thursday August 13, 2020 @05:41PM (#24149745)
> I have to wonder just how relevant all of this effort is anymore

Note that this is ARM32, which is more or less a legacy platform, raising the concern of whether compression is relevant anymore is irrelevant since it's legacy already. As the article says,

> ARM __traditionally__ uses compressed kernels. This is intended as a comprehensive rundown of how the Linux kernel self-decompresses on ARM 32-bit __legacy systems__.

Re:How the ARM32 Linux kernel decompresses by voltagex_ on Thursday August 13, 2020 @07:11PM (#24150846)
Legacy, but how many devices do you think are still functioning with them now? How many are sitting on store shelves?

I wonder if the Google Home next to me is aarch64. The "new" router that is sending my packets to this site is MIPS on an even more ancient software stack.

Re:How the ARM32 Linux kernel decompresses by segfaultbuserr on Friday August 14, 2020 @03:06AM (#24154722)
This doesn't change my argument. Legacy can be interpreted as something that will be phased out in the future, but if you want, you can also think of it as the established status quo, which is hard to change. Whether the compressing scheme should be removed is of no interests - if it's not broken, don't fix it, just keep it as is. If a new ARM32 system is to be designed, it's likely that the earlier limitations such as Flash space is still a factor, so the compression is here to stay.
Re:How the ARM32 Linux kernel decompresses by Koshkin on Thursday August 13, 2020 @08:01PM (#24151494)
I think 32-bit microcontrollers will be with us for a long, long time.
Re:How the ARM32 Linux kernel decompresses by segfaultbuserr on Friday August 14, 2020 @03:06AM (#24154725)
Not disagreeing, I'm still using 8-bit microcontrollers in my projects and I'm quite satisfied on their ability of doing their intended jobs.