> Key-based auth doesn't have this flaw so it's arguably safer in that it has one less entry point.

This is exactly what people think when reading the guides. In fact, it's not that there is one less entry point, but this entry point was moved and now exists on another machine together with the key. And since many people use one workstation to log in to several servers, you now have one single point of failure (an equivalent of using one password for all servers).