If you have CEC or ARC setup, then you can adjust the volume by tapping on the buttons on the Lock Screen or using the iPhone volume buttons when the remote open.

I'm not sure what remote your are using, but the one built into the iPhones pull down menu is instant for me.