Does your synology utilize BTRFS? If so, what is your experience with it? I'm currently using a FreeNAS box that runs on ZFS that has been very reliable for me over the years. I can't really see giving up the power/reliability of ZFS to move to a synology setup, but I like to keep tabs on BTRFS progress.
As for the firewall, I had a Ubiquiti Security Gateway for a while and it worked great. Not sure how it compares to the UniFi Dream Machine, however. The only reason I made a custom firewall is because I wanted to try and learn a bit about firewalls and routing in general. And, since my own device has worked well for me so far, I haven't bothered to replace it.
I can't easily create new VMs, create snapshotted/linked clones, and I can't easily pull out CPU/RAM/disk utilisation information from ESXi for the hypervisor and all VMs to store in a time series database for monitoring/alerting purposes.
There's also a tonne of additional features in Proxmox that requires additional licensing for ESXi (and yes, I am aware of the VMUG EE licensing making a lot of that a lot more affordable)
Proxmox ultimately being Debian based gives a lot more flexibility in that regard. It's also a potential weakness as a result, depending on perspective.
About building your own NAS to replace the synology, take it from someone who has done this, don’t bother.
While rather easy to do, Synology provides so much more “out of the box”. My own NAS ran well on Debian 10 for a few years, until I started getting random disconnects on drives and various other “stability” issues. Synology costs more up front, but after initial configuration it’s more or less just a box in the corner you forget about. Mine just sits there, automatically installing software updates whenever they’re available.
As for my own current setup, I run my hypervisor (Proxmox) on a Dell PowerEdge T30. It runs an internal docker host (Debian 10) and an external FreeBSD host on a DMZ vlan for anything that is accessible from the internet. Everything on that box runs in its own jail.
All storage except OS storage is mounted from the Synology via kerberized NFSv4.
Firewall is handled by a Netgate SG-3100 running Pfsense, but I’m in the process of migrating to a UniFi Dream Machine instead. Pfsense has been good to me, but the SG-3100 is expensive (for what it delivers), and its beginning to show its age. I have a 300/300 mbit connection, and with Suricata enabled I get frequent reboots because it uses too much CPU, causing the watchdog to think its stuck. The UDM is half price and twice the hardware, and for my usage (router/firewall/vpn/dns blocking) the UDM does it all (Pi-hole through a 3rd party solution).