Supporting a JS-disabled experience should not be your first priority. But, reducing needless reliance on javascript will improve your site for every user - making the page load / respond faster, use less CPU and memory, etc. It will also improve the development experience for you - faster to test, less crazy codebase to work in. It's really just architecting the site better.

Some people are so scared of "premature optimization" and "catering to a tiny number of geeks" that they trick themselves into making a huge JS mess. Most websites these days seem to be examples of this.