Features like category lists are implemented via database queries. There's little wrong with that; what's wrong is that small result sets can't easily be pinned in memory.

As with website software, cache performance probably has a great deal to do with how WordPress performs.