It's the same language you already know, but you're missing out on one of the primary ways people use it.
For example the sign languages spoken in the US and in the UK have different ancestries and are not mutually comprehensible, despite both countries using english as spoken languages.
Sign languages have multiple articulators: two hands, face, eye gaze direction, shoulders, trunk. These can all work together to show multiple things at the same time. Spoken languages can really do only one thing at a time (with a few minor suprasegmentals such as tone).
You can construct a signed version of a spoken language, which may be useful for things like quoting book titles and other cases where you need to represent the exact words of a spoken language in signed form, but it's not common to use that for everyday communication, because the hands move a lot slower than the small muscles of the mouth and throat.
(Linguistics is a fascination of mine. Sign language linguistics are especially interesting.)
Well, yeah. I guess my point is that ASL is pretty much a foreign language.
I'd compare it to the situation with the english language worldwide - since english is the lingua franca, so to speak, many countries around the world teach it as a second language. If you don't learn english, then (generally speaking) you're at a disadvantage because you can only communicate with a subset of your population.
I'm not saying there's a problem with deaf people, any more than there's a problem with anyone else who simply doesn't happen to know the languages of some of the people around them.