Airbnb's preferred smart lock vendor accidentally bricks 500 door-locks

Re:Airbnb's preferred smart lock vendor accidentally bricks ... by cperciva on Monday August 14, 2017 @02:43PM (#15011320)

we had a scheme where we'd push an update, it'd reboot, and if it didn't connect to our server within 20 minutes it'd revert to the previous version and reboot again

Was this handled in userland? Or with something lower level (e.g., a hardware watchdog)?

I'm asking because the question has come up a few times recently about making FreeBSD recover (unattended) in the event that an unbootable kernel is installed.

[ Parent | on HN ]

Re:Airbnb's preferred smart lock vendor accidentally bricks ... by tlb on Monday August 14, 2017 @03:00PM (#15011544)

Userland. Basically during upgrades we installed a shell script in /etc/init.d:

  sleep 1200
  if ping -c 5 central-services.anybots.com; then
    rm <this file>
  else
    cd /home/dist && git checkout STABLE && make install
    reboot
  fi

(We used git to distribute binaries, so we could checkout and install a previous version easily).

This was a last ditch recovery. I could imagine lots of things that break, but which still allow ping. And installing a previous version over a newer version is flaky, because a newly created file won't be deleted.

An unbootable kernel is a whole different kettle of fish. I believe GRUB can be told to fallback to a safe mode.

[ Parent | on HN ]

Re:Airbnb's preferred smart lock vendor accidentally bricks ... by cperciva on Monday August 14, 2017 @03:15PM (#15011764)

I believe GRUB can be told to fallback to a safe mode.

I'd assume so; FreeBSD's loader can also do that. The problem was that if a kernel hanged during boot, we wouldn't get back to the loader until someone power cycled the system, so the "try the other kernel" logic wouldn't run.

[ Parent | on HN ]

Re:Airbnb's preferred smart lock vendor accidentally bricks ... by srcmap on Monday August 14, 2017 @03:59PM (#15012266)

This will be the time where setup HW watch dog should help.

After the upgrade, before the reboot, setup the HW watch dog for 2-5 minutes. Setup the boot-old-kernel flags as default in the uboot/EFI/grub/or your system boot loader.

If the upgrade the successful (connect to back to the server) and function correctly, disable the "boot-old-kernel" flags in your bootloader.

As always, do a lot of testing and automate those tests. You will be surprise how often a simple boot/reset/power cycle can fail if you repeat it over and over again in a weekend.

[ Parent | on HN ]

Re:Airbnb's preferred smart lock vendor accidentally bricks ... by cperciva on Monday August 14, 2017 @04:02PM (#15012291)

Hmm, you can set a watchdog before rebooting? I assumed that it would be cleared automatically.

[ Parent | on HN ]

Re:Airbnb's preferred smart lock vendor accidentally bricks ... by joezydeco on Monday August 14, 2017 @04:01PM (#15012280)

U-Boot has a mechanism for offering fallback by counting boots (and saving that to local EEPROM).

If you set up your watchdog correctly and the system loops too many times, your u-boot launch script can swap booting to safe/backup partition.

[ Parent | on HN ]

Re:Airbnb's preferred smart lock vendor accidentally bricks ... by swiley on Monday August 14, 2017 @06:06PM (#15013447)

It's fun to poke around in these embeded firmwares. Most of them are strung together with shell scripts for everything but the actual application, this kind of stuff is generally just another script.

[ Parent | on HN ]