Key escrow has a number of problems, not the least enforcement that keys are valid and validated. (something that there's not a good history of, and international issues come up)

Back doors are worse though - build a back door and it will be used, just not necessarily by the agency it was built for. There are a lot of groups with a lot of resources oriented around taking advantage of this, and few are legitimate. (and some are enemy nations).

There's a third problem - doing it in such a way that it can't be blocked from monitoring. (see "clipper chip" for more on that).