The UIs of the OSes designed in the 640x480 era used 8pt system fonts, and were very careful about "spending" vertical space. The UIs of the OSes of today are not optimized for the same scenarios.

Here's what Windows XP (not even really from the 640x480 era!) looks like on a modern display: https://ibb.co/W5mVgHT

You could fit a lot of Windows XP, and XP-era apps, on a 1366x768 display. You can't fit much of a modern OS+apps on one.

(That's honestly for the best; text and icons in modern OSes are both a lot more legible due to the increased fidelity and breathing room they have. But display resolutions have to keep up; and, at least for low-end PCs, they aren't.)

What you're showing here is lack of high DPI scaling and it's a completely different issue irrelevant to the GP's point - it's caused by increased pixel density, not screen space (unless your example comes from some 40" monitor or so). To make an honest comparison you would have to set the output resolution to a lower value.

When done right, increasing screen density doesn't really affect available space, just makes everything sharper (unless you manually reconfigure it to get more space, like I do on my hidpi laptop).

Okay, maybe running XP natively on a 4K display was a bit silly. But here's XP in 1080p: https://ibb.co/BB5rsHg

UI elements are still far smaller than comparable ones in Windows 10 or macOS, when those are running at 1080p unscaled. On XP, IE and Explorer both manage to fit four toolbars (rather, a title bar + three toolbars) in 105px of vertical space. On my own macOS Catalina install, Chrome manages to fit only two toolbars in 86 device-independent pixels of vertical space.

Likewise, just consider the fact that people weren't generally looking at PDFs back in the XP era, but rather plaintext, which—as rendered on this image—was exactly 8px tall. A lot of 8px-tall text fits on a 1080p display, no matter how many toolbars you have; and a lot of 8px-tall text will even fit on a 640x480 display, if you make efficient use of the vertical space. But because PDFs use vector fonts, not bitmap fonts, they will look unreadably muddy if you attempt to scale one down such that the text only uses 8px of vertical height. You need more vertical space to look at PDFs.

So, in combination, when you throw a Windows 10, running Google Chrome, displaying a PDF, on a 768p display, it's just an ugly constrained problem.

As mentioned the dpi reasons you described are somewhat unrelated. But there is one factor, the move from 4:3 to 16:9 ratio monitors, which is less efficient in terms of usable area.

Given that most scrollbars are vertical however, it isn't a big deal. I've gotten more vertical space from putting my taskbar on the left (when 16:9 became ubiquitous) than anything other change.

Any operating system would look like that on such a large resolution.