Indeed, other hospital chains do as well, which is why we viewed it as a good option and went down that path to begin with. In the case I'm referring to, everyone at the hospital already had badges and the thought was that removing password requirements and using the badges that everyone already had as a login would work well.

It didn't work, not because of technical issues, but because we didn't anticipate the high number of doctors that apparently had lost their badges and had never faced consequences for it (the culture at this hospital was "oh you forgot your badge? no worries, I'll just open the door for you"). When we then asked the medical staff to keep better track of their badges (not just for the login system but also because of general campus security) we received incredible pushback, and that's when we had to roll back the program.

IME, and as evidenced by the VA using a similar system as you mentioned, doctors are perfectly competent enough and able to use these systems and do just fine once they get used to the system. The issue is that they put up a fight more than anyone else when introducing something new, and oftentimes IME the new system never gets a chance before it's shot down.