It would be naive to download and run native code for every website you visiti, yes. A few that you trust and where you think that is warranted is a different matter.
Running javascript in a sandbox provides the illusion of safety so it gets enabled by default while still creating tons of problems.
We tried that, and the security issues it caused were orders of magnitude more severe than any of the problems caused by defects in an up-to-date browser sandbox. Basically all consumer PCs used to be infested with viruses all the time. It sparked an entire virus scanning industry.
It takes far too much discipline and diligence to make sure that you can trust the motivations and security capabilties of all your software providers. Sandboxing is good. It's the only thing short of the most heavy handed, restrictive and centralised control that has ever worked.
The security issues we have on today's Web are overwhelmingly unrelated to client-side security. The problem is protecting the data that is stored on servers and the incentives created by ad based business models. All of that is equally problematic regardless of whether you run native code or sandboxed JavaScript.
The browser runtime is what enabled us to use software provided by a huge number of developers of varying aptitude and motivation without putting in place some centralised gatekeeper with its own vested interests.