For autocompletion, datepickers, etc I use drop-in components. These tend to be jQuery based but it's not the end of the world.
When I do have a particular view or flow that outgrows the "bits and pieces of JS" phase it's actually pretty straightforward to use a modern UI library like React or Elm on that view alone. Reducing the scope of the area you're using the framework in rather than writing the entire app in it simplifies things pretty dramatically. The JS doesn't need to worry about routing, pushState, etc etc etc. It just deals with its own little patch of DOM and state.
There was a standard (https://tools.ietf.org/html/rfc3229) to diff pages so that only the delta would be sent to the browser. After that the only missing piece would have been that the browser doesn't refresh the whole page but intelligently re-renders the difference in the DOM, and that would have been the perfect world.
Of course, this is just wholesale replacing a DOM subtree with another. But there's also probably some point where just parsing the new subtree is faster than attempting to create and apply a diff.
I'm all for server side rendering - I maintained several vanilla PHP/HTML/CSS websites in the 2000s, and only used jQuery sparingly when things grew in complexity. I later used Django for many years similarly, and when react/ember/angular/etc. started blowing up is when I started feeling like performance was something we had to worry more about.
That being said, server side rendering shows its limits when you build an application with lots of actions your users can perform on the data shown in their window, and reloading the page every time would significantly slow them down. That's when you're tempted to start going the SPA route, but those frameworks tend to be pretty opinionated about how you should structure your code and data, a slippery slope that leads to the stereotypical bloated, unresponsive JavaScript mess some websites have become famous for.
There is certainly a balance to be found, but it's a tricky one.