A lot of the grammar is extraneous indeed, and not actually used in spoken language. After taking a linguistics course, I started paying more attention to the difference between my spoken and written Québec French (my mother tongue) and realized that we almost never use the formal future tense in our speech; we use an auxiliary instead, English-style. So instead of saying "J'irai au dépanneur", I say "Je vais aller au dépanneur" (I will go to the convenience store). We often do the same with the past tense (passé composé), although that form is more kosher in written form.
When a speaker slows or halts a sentence because he/she is struggling on grammar, it makes it exponentially harder for the listener to keep listening and interpret what is said. On the contrary, grammar is so redundant that getting it wrong on half of the words would still leave enough information to understand even the nuances.