Was this handled in userland? Or with something lower level (e.g., a hardware watchdog)?
I'm asking because the question has come up a few times recently about making FreeBSD recover (unattended) in the event that an unbootable kernel is installed.
sleep 1200
if ping -c 5 central-services.anybots.com; then
rm <this file>
else
cd /home/dist && git checkout STABLE && make install
reboot
fi
This was a last ditch recovery. I could imagine lots of things that break, but which still allow ping. And installing a previous version over a newer version is flaky, because a newly created file won't be deleted.
An unbootable kernel is a whole different kettle of fish. I believe GRUB can be told to fallback to a safe mode.
I'd assume so; FreeBSD's loader can also do that. The problem was that if a kernel hanged during boot, we wouldn't get back to the loader until someone power cycled the system, so the "try the other kernel" logic wouldn't run.
After the upgrade, before the reboot, setup the HW watch dog for 2-5 minutes. Setup the boot-old-kernel flags as default in the uboot/EFI/grub/or your system boot loader.
If the upgrade the successful (connect to back to the server) and function correctly, disable the "boot-old-kernel" flags in your bootloader.
As always, do a lot of testing and automate those tests. You will be surprise how often a simple boot/reset/power cycle can fail if you repeat it over and over again in a weekend.
If you set up your watchdog correctly and the system loops too many times, your u-boot launch script can swap booting to safe/backup partition.
1. A backup recovery read-only image which can flash everything back to square zero via some sort of hardware over-ride (like a pin-hole button that has to be held down for 10 seconds)
2. A temporary place in memory to buffer and validate the checksum of any new firmware that is uploaded before flashing it.
Real time flashing over spotty WiFi (without any sort of post-transfer checksum validation) has got to be the dumbest, laziest, cheapest, riskiest thing a hardware manufacturer can put out, and any manufacturer that does that should be ashamed of themselves.
I can also confirm the wifi problem. Not knowing if a device is bricked or just on bad wifi is a huge problem you have to somehow account for. It was terrifying before I built tools to keep track of which devices those are
Practically, there are tons of instances where executing code downloaded from URL's is how software is updated, but again most of those instances are human-driven, not automated.
If someone gains access to our server, its already game over. Being able to execute code on the devices is then moot because we were already hacked & potentially lost all credibility in the industry.
To everyone saying not to execute code at URL, this is how oh my zsh, and many other open source projects are installed. And SSL should prevent any kind of MITM attacks. The point about keeping keys off the server is a good idea, but is not always possible when the server needs to dynamically send content down to the device. The server would need to access the private key in order to sign this content dynamically. For statically built executables this is a good idea though.
Just in case the domain name lapses. (Or for any of a hundred other reasons why having a system that reads plain text commands from the network is a bad idea)
The scariest times were pushing updates that affected WiFi connection for customers with unusual network configurations. In particular, we had one with a multi-campus Cisco WiFi system with WPA-Enterprise, which requires logging into a RADIUS server to get WiFi. We were running on FreeBSD which had a half-assed port of the Linux WPA authenticator. We never managed to replicate their network perfectly for in-house testing, so pushing updates to them was always nerve-wracking.