Some users don't know how to use their browser's zoom feature. Most don't know how to change the default font in their browser, and those that do might not want to do this in case some sites break because of bad assumptions. Others don't like zoom, because it makes images look blurry. The default appearance of the page should be usable and accessible by an average user.

And yes, the 20px font applies to the full text. I'm not arguing that smaller fonts should be banned, things like metadata don't need to be as large. But the main content of the page should be larger and comfortably readable.