The flu probably wouldn't be dramatically worse if we didn't have a vaccine. The vaccine isn't particularly effective all things considered, ranging from 10% to 60% depending on the year and how well scientists were able to guess which strain would be predominant.

The flu as-is causes 45,000,000 sicknesses every year in the US alone.