Keep in mind that this is a translation - originally, Einstein wrote this in German [1].

most rational thinkers understand that basic defense is necessary

I think he is looking at things from the perspective of the institutions' interaction with individuals - in particular, that those individuals subjugated to the military or similar institutions are an enormous waste of potential. This part, in particular, speaks to that notion:

"He who joyfully marches to music, rank and file, has already earned my contempt. He has been given a large brain by mistake, since for him the spinal cord would surely suffice."

1. Or at least, there are slightly different versions out there: http://rescomp.stanford.edu/~cheshire/EinsteinQuotes.html