I think a simple linux box with docker/kvm/proxmox would be quite sufficient to start with. You can just add bridges between VMs / Containers,... advance to SDN, setup your own Firewalls and LoadBalancing VMs... and for watching network traffic you can use wireshark/tcpdump.or whatever, you csn even plug in monitoring or tracing proxies in between. When using LXC containers with proxmox the system should be quite sleek.

I would recommend to stay as open source as you can. (using pfsense, nginx... not commercial firewalls), play around with wireguard,... also use a IaC configuration tool like ansible or pyinfra do configure your stuff from.the beginning (so you can easily reconfigure stuff and make changes in scale)

Have fun!