How would likes, retweets, and replies (that the original author can be notified of) work with that architecture? ActivityPub and the protocols that predate it were worked on for quite some time, it seems unlikely that nobody would've noticed "oh, we could just do it all through RSS" if it were indeed that trivial.

Also, the model of Mastodon is more like email -- you are going to have far more clients (as in, several orders of magnitude more) than servers, so getting clients to do those fetches isn't better for the health of the network. ActivityPub (when it comes to server-to-server federation) is mostly push based as far as I understand.

I think the primary issue is that the primary Mastodon server implementation wasn't written in a framework renowned for scalability.