Reading between the lines, I think what they would need is a way to tell clients to move to a new websocket connection at the proxy layer. I don't think there is an easy built-in way to do this in the websocket protocol, so they would have to implement something custom in their application layer. This would also require triggering custom code in the client to make a new websocket connection, start using it, and then close the old connection.
I feel like it would have been simpler to just have the client do a graceful reconnect every 5 minutes. But they probably decided to use envoy so they could have the other advantages too.
If you do restart with reconnect, many clients will need to reconnect multiple times, as it they get kicked off early, they'll tend to connect to another old server, and have to get kicked off again later. This isn't great because connection startup is usually expensive; also a (hopefully) small minority of clients may have difficculty connecting because of network issues and it may take them a substantial time to get reconnected.
If you have a working hotload path, you get clients to new server server code faster using less resources.
Load balancing software is also often behind some other layer of load balancing/availability (DNS or ECMP or BGP or CARP), and a hot load means you don't need to churn that layer, which can avoid any issues with changes in that layer.
Dependents on how many config changes you need (per day).
Besides Envoy supports it and I would call it a bonus if you can reload your configuration without client interruption. As for complicating things the implementation for hot reload isn't terribly complicated in Envoy.
This "drain" pattern is great for maintenance, upgrades, etc too.
The only caveat is that the clients need to be given time to migrate. How long that is depends on how well the clients behave. A hot restart may be much faster.
> Consul-template renders a new version of the host list, and a separate program developed at Slack, haproxy-server-state-management, reads that host list and uses the HAProxy Runtime API to update the HAProxy state. ... However, over the course of the day, a problem developed. The program which synced the host list generated by consul template with the HAProxy server state had a bug.