We run a stupidly simple (though small scale) gitlab instance in docker (official image its kinda one-click install). You get mattermost automatically, just need to enable it in the gitlab config and it will install/run mattermost within the gitlab container without any troubles.
So if you already have (self-hosted) gitlab, mattermost is easy to setup and maintain (actually no extra effort at all). Depends of course on the number of users, I guess...
The integrations with gitlab issues/groups/etc are also look neat but are barely used tbh. Cant compare with rocket.chat.
That said, personally i use Rocket.Chat, which has been pretty easy to self host: https://rocket.chat/
I'm sure that someone who has experience with both could drop by and share their experience, but if nothing else, it's at least nice to have these options to choose from.
Either way, i can definitely confirm that most of these platforms indeed are very well off on smaller VPSes, as long as you don't have too much data in them (think file uploads rather than just messages).
Getting more support for something like that in the company that i work in, however, is another story entirely. For a while they tried Nextcloud Talk, but it was a bit too barebones: https://nextcloud.com/talk/
And ever since, sadly, there has been little to no interest in setting up another self-hosted platform, even if there are teams whose old messages are eaten away by Slack's limitations, the management seemingly having little to no interest in paying a bit of money for it.