Effectively nothing can be constrained by a whitelisting firewall if you have a sufficiently bored actor. You can smuggle data through a variety of benign looking protocols, things that wouldn't matter in the least generally. Your average server contacts hundreds of different public NTP servers, binary repositories, domain name servers every day. If the keys to the kingdom are a 32 byte ECDSA private key, you've lost if you think you can protect this from reaching the outside world.

A method that wouldn't show up on any firewall in the world is simply to delay or drop certain SYN packets. Even if you only intended to transmit a bit at a time through this, any unauthenticated host on the internet could use this without raising any suspicion or even printing log lines in most environments. As soon are you're making an assumption that you're trying to prevent what's inside from getting out things become substantially closer to impossible than anybody would like.