Never use a Turing-complete language to solve a problem that's already well-solved with a non-Turing-complete language. Especially when that non-Turing-complete language is already present on the client's machine. Most especially when you're talking about static sites for essential public information.
On the other hand, if you need a site to work even with spotty internet access, JavaScript is probably your answer.
But these are generalities, you can probably find a thousand exceptions. It is hard to give truly general advice because there are so many exceptions.
That said, there are many good reasons to have static sites when you have to scale up big. They typically can handle very large scales, and caching them the super easy.
Whether or not that actually meets the requirements of a site is a completely different story.
> get it down to static HTML and CSS and maybe a tiny bit of enhancing JS, and pare away every byte you can.
HTML is a language used for building your DOM. CSS is a language used for styling. JavaScript is also a language that can do those things, and focusing on the language used isn't going to help.
If the goal is to save space and power you can do it much better with JS. A lot of server side rendering, html caches, and static content is focused on good SEO and low latency, not small download size and power consumption.
You can fit a much smaller download into a stripped-down Gatsby site if you have a nontrivial amount of content. It also makes things like serving only much smaller versions of any images way easier. You can go even smaller with vanilla JS if you want.