About a year ago was in many places just after the start of the pandemic, which brought a large influx of users to the default public server. It got overloaded, and slowed down. I experienced it, too.
They have made several rounds of significant scaling improvements since then. It's almost never slow for me any more. (And even if it was, I could always just choose a server other than matrix.org, or run my own.)
That was probably a problem with their main matrix.org homeserver, more than with the protocol or your particular client app. It was very slow for a while, until they worked out some scaling issues to handle the load.
I’m on matrix.org pretty much every day now, and it’s a lot better than it was. And when I use my own server to talk with other local users, I never have any latency issues.
Has this changed recently? I would like to give it another shot as I want to support the tech. Can someone with use experience please comment with their experience on the speed?