I’ve had the same concern for a long time, but more from a worry about what to do when it breaks or stops working after a few years. The recommendations I’ve read are to always have a recovery key or other alternate mechanisms that don’t depend on this device. One common suggestion also seems to be to print the recovery/alternate key and keep it safe.

I believe this is a bigger barrier for common people who don’t know how to create and retain backup mechanisms. As such, I don’t see a lot of value in recommending these devices to those who aren’t tech savvy without also explaining to them about recovery. So much for technology!