I make the same association. A door is part of a building, and is put there during construction (initial release) or in owner-planned renovations (software updates).

I've never heard of someone breaking into a building by cutting a hole into a wall to install their own entry door that they have a key to, but that's the scenario this "OS X backdoor" is describing.