I'm running on an Arch Linux based home server, what in particular gave you trouble?

That's my exact setup. Works flawlessly.

Do you have a domain, with a valid https cert? Chromecast needs to make a couple requests, and will only do it over https, and requires the cert to be valid.

If you’re running over http, or using a self-signed cert, Chromecast won’t work.

Chromecast works reliably here (served from Debian). DLNA is broken, though, but that's probably a firewall rule.

It took me a while to realize that Chromecast doesn't play from the phone itself, you need a server with a valid HTTPS certificate that's reachable from the Chromecast itself to play content. You can also use plain HTTP, as long as you don't use self-signed certificates. For the routing problem there's no solution, you can't, say, have a phone hooked up to a VPN and play to a Chromecast on the local WiFi.

There's also an issue with putting Jellyfin in a different subdirectory (i.e. videoplayer.local/servers/jellyfin) but for me that still seems to work, luckily.

Also make sure the Chromecast can resolve your domain through its DNS resolver. If you use a local DNS server (i.e. the one built into your router) to resolve hosts in the LAN, you need to block 8.8.8.8 and 8.8.4.4 so Chromecast doesn't have a choice but to use your normal DNS server.

Firefox also doesn't seem to like my Chromecast but that's the case for every video I play.