From what I understand, the audio and video are simply different streams that you can selectively download from the youtube servers. So it is entirely possible to only download the audio, I've done it many times with youtube-dl.