I've never worked with a CMS or web app - in any language - that didn't include caching as a core part of the performance layer once it reaches a certain amount of load..
I'm sure it's possible, with enough money to throw at hardware, to scale without it, but why would you? Caching is a completely valid optimization approach in my view..
We should not be needing bare metal servers with 4 GHz CPUs to serve a non-interactive site with ~4000 visitors a day. Yet, without full-page caching (i.e., bypass Wordpress/PHP entirely and serve .html files straight from nginx), the servers would melt under the load.
Rendering twenty images and forty text fields out of database should not take several seconds on a modern server. Yet, Wordpress has no problems taking a dozen seconds even under light load conditions – mind you, that's time purely spent in PHP, for rendering, not on the database.
I'm not making any specific value statement about your setup here, I'm just surprised by what you're seeing on your end.
Still, we're usually seeing a lot better performance with everything that's not wordpress – be it our in-house PHP framework, or node, Rails or Python apps.
Sidenote: why doesn't the mysqld configure script suggest various performance levels upon setup, based on detected hardware (RAM, disk, CPU cores)?
Both can be plugged into memcache and friends, and if done right you can put in Cloudflare or other frontend caching systems in front.
Biggest performance +1, though, is enabling php opcache.
Well, these two platforms are by far the leader in the CMS space. It's either usability, ease of administration or speed that keeps people using WP/Drupal over alternatives in other languages/frameworks.