And before you say that DD-WRT is not $WEIRD_CHINESE_BRAND and obviously doesn't have such vulnerabilities on its HTTP interface that can be accessed without auth, see https://www.cvedetails.com/vulnerability-list/vendor_id-9341... for some real examples, including "getting root by sending an HTTP query that doesn't require any authentication".
In fact, for that remote root exploit, disabling Javascript execution wouldn't even help that much, because it can be triggered through e.g. <img src="..."/> on any HTML page you visit (with some guessing required for the IP address, but there are only so many IPs people commonly use for their gateway).
You should still apply security fixes for (relevant) kernel and sshd issues, of course.
Having security updates pushed to your phone is never a bad thing.
It's the reason why enterprise Linux distro updates promise security patches (but possibly not bug fixes) for such a long period - because people do care about these things. Security patches are a good thing.
My router has all WAN-side management features etc. turned off, WPS is turned off and it has a long random WPA passphrase. Unless someone breaks into my apartment, there just isn't a significant potential for security risks that I'm aware of.