> Also, JS is cached. First time will be slow, others, not so much

This is a very weak argument.

  * many 3rd party resources are not cacheable or may have bad policies
  * caches may be more often cold then you expect. most apps still cache large combined blobs, which will invalidate with each release.
  * pure JS size has still it's impact on performance
  * caching won't magically cut all network roundtrips which you need due to architectural reasons

Yes, but you can optimize your website for caching. Your points are valid but they mostly boil down to "people don't know/don't care what they're doing", which won't help even if you don't have JS in your site.

And it is: "still its impact"