Instead of directly exposing the server to the internet, you could keep it behind a VPN. That way you only need to keep the VPN secure, the VPN serves as a whitelist.