My phone controls the volume, when the remote app is up on the phone the volume buttons on the phone actually change my receivers volume through HDMI from the Apple tv.

This depends on HDMI-CEC and volume control support amongst manufacturers is varying. My fancy LG OLED doesn't support volume control over CEC (but it has HomeKit support which works).

You're the second person to say this, but for what it's worth: the iPhone remote app is literally the only way I control the volume on my AV system.

First, let me just say that it never even occurred to me that my iPhone volume would control the TV so I just never even tried it. I would have expected volume buttons to appear in the remote app.

That said, I just tried it and it does not seem to work for me in my setup. I just have a Samsung TV with the Apple TV connected. No other "AV system". The volume buttons do not seem to do anything with the TV. Now that I know it works in some situations I will start digging around.

The Apple TV remote controls the volume of the TV fine but I seem to recall that I programmed it for the power and volume of my TV when I set it up.

I have a specific annoyance about that: I have an Apple TV HD that I bought second hand that had no remote; it's connected to a monitor so there's no TV remote or HDMI-CEC or anything of the sort; when using AirPods with the Apple TV the only means I have to adjust volume is that iOS App, the universal remote I have for otherwise controlling the Apple TV can't adjust it. Oh well. I'm thinking about buying that new remote and taking one of the other siri remotes I already have and putting it on that one.

Me too mostly but it's annoying that it's in a weird place not directly accessible from the remote widget in control center.

For me, using both the "built-in" remote app on the control panel and the "real" remote app launched as a proper app, the volume is controlled simply with the iPhone's volume buttons. I'm not sure how it could get any more ergonomic.