You can bind a device to another, so while you would need the ability to issue the command, a server wouldn't be required to handle the state propagation.

https://smarthomescene.com/guides/how-to-bind-zigbee-devices...

"mostly zigbee?" not sure what you mean there. But ESPHome can be controlled directly without HA. You should read the website, specifically the sections on "Networking" and "Management and Monitoring".

If you are starting at zero there is a big learning curve, but if you're into it, it is a lot of fun.

Why do you want to avoid Home Assistant? From what I've found setting up my automations is that once you have it doing something useful you find you have a lot of ideas to further make your life better. I might be wrong but I suspect you'll spend a lot of time doing something that is simple in HA only then find you want to do something else similar.