Airbnb's preferred smart lock vendor accidentally bricks 500 door-locks

Re:Airbnb's preferred smart lock vendor accidentally bricks ... by tlb on Monday August 14, 2017 @03:00PM (#15011544)

Userland. Basically during upgrades we installed a shell script in /etc/init.d:

  sleep 1200
  if ping -c 5 central-services.anybots.com; then
    rm <this file>
  else
    cd /home/dist && git checkout STABLE && make install
    reboot
  fi

(We used git to distribute binaries, so we could checkout and install a previous version easily).

This was a last ditch recovery. I could imagine lots of things that break, but which still allow ping. And installing a previous version over a newer version is flaky, because a newly created file won't be deleted.

An unbootable kernel is a whole different kettle of fish. I believe GRUB can be told to fallback to a safe mode.

[ Parent | on HN ]

Re:Airbnb's preferred smart lock vendor accidentally bricks ... by cperciva on Monday August 14, 2017 @03:15PM (#15011764)

I believe GRUB can be told to fallback to a safe mode.

I'd assume so; FreeBSD's loader can also do that. The problem was that if a kernel hanged during boot, we wouldn't get back to the loader until someone power cycled the system, so the "try the other kernel" logic wouldn't run.

[ Parent | on HN ]

Re:Airbnb's preferred smart lock vendor accidentally bricks ... by srcmap on Monday August 14, 2017 @03:59PM (#15012266)

This will be the time where setup HW watch dog should help.

After the upgrade, before the reboot, setup the HW watch dog for 2-5 minutes. Setup the boot-old-kernel flags as default in the uboot/EFI/grub/or your system boot loader.

If the upgrade the successful (connect to back to the server) and function correctly, disable the "boot-old-kernel" flags in your bootloader.

As always, do a lot of testing and automate those tests. You will be surprise how often a simple boot/reset/power cycle can fail if you repeat it over and over again in a weekend.

[ Parent | on HN ]

Re:Airbnb's preferred smart lock vendor accidentally bricks ... by cperciva on Monday August 14, 2017 @04:02PM (#15012291)

Hmm, you can set a watchdog before rebooting? I assumed that it would be cleared automatically.

[ Parent | on HN ]

Re:Airbnb's preferred smart lock vendor accidentally bricks ... by joezydeco on Monday August 14, 2017 @04:01PM (#15012280)

U-Boot has a mechanism for offering fallback by counting boots (and saving that to local EEPROM).

If you set up your watchdog correctly and the system loops too many times, your u-boot launch script can swap booting to safe/backup partition.

[ Parent | on HN ]