It's really a confluence of factors. There are even elements rooted from the specification itself that contribute, for example requiring many round trips in network interactions with a lot of different servers; requiring large amounts of random disk access to implement core features... These things tend to shape the experience born out of client and server implementations even after a fair amount of performance work. I hope that Matrix-org makes an effort to consolidate the specification into something more streamlined for any upcoming versions.

Synapse, the reference impl server, was always pretty slow. It got a lot faster with its Python 3 transition and there is a Go server in development that I think the core team means to focus on once 1.0 is out.

There are also native clients like Fractal and Quaternion that use GTK / Qt UIs that are just as fast as any other native app.