I actually think polyfill usage is backwards: the current approach allows code targetting new features to be made compatible with old browsers, by the site providing polyfills.
I think it's much better for new browsers to be leaner, simpler and more well-thought-out than their predecessors. Such browsers can provide polyfills to allow compatibility with code targetting old features that have been dropped.
In other words, browsers should be getting smaller (e.g. a canvas, audio stream, input method and language interpreter); more and more functionality should be offloaded into interpreted libraries (parsing, text rendering, DOM, event loops, layout, Javascript, etc.)
As a web developer, I much appreciate native features and avoid polyfills as much as possible.