A ten year old can transcribe arbitrarily complicated speech in their native tongue but could have difficulty summarizing it or generating it. There are more tasks to unlock with larger and larger language models, but the language model for figuring out the most likely utterance given the audio doesn't have to be nearly so big, just like a reasonable language model for summarizing can be much smaller than one for dialogue.

Speech-to-text is transcriptive, it spends its parameters on encoding a mapping from one domain to another. Language models are generative, they spend their parameters on trying to encode a domain. It might seem like speech-to-text should need more because it cares about two domains (“speech” and “text”) while language models only care about one domain (“text”), but it turns out mappings are much smaller than domains.