It's as if there were a programming language that greatly improved your productivity, but was only available for SPARC boxes running Solaris. Sure, the few times you were able to use such a system, it would be great. But how many programmers would bother to learn a language that they would use so infrequently?
The point is, ASL must be a language first, and an aid for the disabled second. On that point, it fails. The goal of a language is to enable communication; if less than 1% of people you are likely to interact with are capable of communicating in that language then it is of limited utility. People who have normal hearing are generally not going to learn ASL, as the value proposition just isn't strong enough. Because of that, even people who have hearing problems might not be incentivized to learn it, for the same reason: it just might not be that useful.
Now, as with any group, a shared language leads to shared culture and a sense of belonging. That, in and of itself, will make ASL attractive to many people. That's fine. It fails at its intended use as a primary means of communication however, as it just doesn't let you communicate with that many people. This, I think, is why many deaf people also learn to lipread: it allows for communication outside of the (relatively) small deaf community.
http://www.hearingconnection.org/hearing_loss_statistics.htm...
2% is actually a little generous based on this data, but I like to round up. 35 to 36 million people report some degree of hearing loss, out of around 311 million people in the US, or about 10% of the population. In 1972, an estimated 642,000 people signed ASL at home, and about 375,000 of those were deaf. It's somewhat likely that the number of signers has gone down since then, due to the decrease in rubella (a significant cause of childhood deafness), the oral deaf and cochlear implant movements, et cetera. It's unlikely that the number of signers has increased. So assuming there are still around 375,000 deaf/HoH signers in the country, that's 375,000 out of 35 million people with hearing loss, or just about 1%. I put the extra percent in there for wiggle room.