I think it's part ignorance, part exceptionalism. Backdoors sound simple, and if you're thinking about physical backdoors people are generally pretty good at protecting them. That this is largely because they have a lot of characteristics not shared by digital backdoors is easily lost on most people. These folks also tend to believe that THEY will be perfect stewards of backdoors, and anybody who loses control of them is just less competent.