Who do you add as a contributor though? Wannabe-contributors? Then they appear in the list of contributors before you even see if they're capable of producing an acceptable PR.

Your solution would be great if GitHub would also allow me to whitelist specific users, but unfortunately this still won't block "implementation plans" in comments.