You don't want to give unlimited access, but once someone authorizes an individual as a contact, the contact should be kept up to date automatically if a phone number or email address changes.
(Also, partner organizations might want to share directories easily; I can see giving a full access to your directory to a contracting/temp agency you use, rather than giving them accounts. I'm sure there are better examples.)
It sounds like a good idea- I used to think so, myself- but I've steadily been convinced otherwise by the (quite sharp) IT guys managing our directory.