Taking them down also makes them less available to deaf people. If you're deaf, and a video without CC is available that you're interested in, you can get someone who can hear to translate it for you (perhaps in real time into sign language or do the CC work or whatever). If the video is gone, you're out of luck, no matter your auditory capability or lack thereof.

"I can't hear it, so I'm better off if nobody else can either" doesn't really compute.

Note that on YouTube you can open access to a video for community contribution. Someone other than the original uploader can come along and provide closed captioning.