Something like close the connection after N requests/N minutes if the nodes are mismatched, but leave it open indefinitely if they match.
There's of course a lot of ways for that to not be very helpful. You'd still have only a 25% of getting a port number that hashes to the right node the next time (assuming tcp port number is involved at all, if it's just src and dest ips then client connections from the same IP would always hash the same, and that's probably a big portion of your clients), and if establishing connections is expensive enough (or clients aren't good at it) then that's a negative. Also if a stream's files don't tend to stay on the same node, then churning connections to get to the right node doesn't help if the next segment is on a different node. I'm sure there are other scenarios too.
I know some other CDN appliance setups do use multiple IPs, so you probably could get more, but it would add administrative stress.
In a past life we broke LAGs up to use different subnets per port to prevent traffic crossing the NUMA bridge.
I’m sure there are good reasons you didn’t take this approach, be interesting to hear them.