It just needs to be smarter about the context. If a video is loading in chrome, 90% of the time the intent is to turn down the volume of the video, it's broken that it waits for it to start playing.

Honestly, the most intuitive option for me would the audio controls should never touch the ringer, bury that behind a menu, rather than switching based on context.

I haven't used Android in a while but that used to be a setting... "Volume controls:" with a selector for what it did... Volume, ringer or both

How about having them not ever adjusting the ringer?

You can always adjust the ringer (something people rarely do -- for quite times there's the "silent" mode anyway) from the UI, and always use the volume buttons for the system's (music, video etc) audio volume.