You can undoubtedly put priority of grammar over vocabulary, no? You are not thinking in terms of systems that allow us to find the "words" we need after being guided by our mastery of the grammar and ability to reason about what must be said given a situation. I think what you said is valid only if we think about speaking a language unassisted by these types of systems, which is exactly what I feel must be left behind.

Edit:

Also, this is what happens in practice anyway! Doctors don't remember a lot and constantly have to look things up ( as is understandable and expected ). You must be a master of the "grammar of medicine" though. Otherwise you won't be able to construct well formed sentences... aka medical thoughts.

You need some memorization to give some concrete instances of the grammar, of course; however, going overboard on the "vocabulary" is not constructive and shifts the focus onto knowing a lot of words, not knowing how to construct great sentences. It's like a writer that knows a lot of uncommon words, but ends up having nothing meaningful to say... because the power of writing is not in the vocabulary alone, but in the craft of composition.