Works fine for me. As in, I set it to usb audio and it keeps the setting after I restart the computer or resume from suspend. Sometimes I want HDMI audio and then I switch the output using the Gnome volume app and that works as well.

If it's a PulseAudio issue, it's very likely that you can change something in a config file somewhere to change the behaviour to whatever you want.

Dunno, never connected it to HDMI.

I recently switched from pulse/alsa to Pipewire with the Pulse shim, working pretty well so far. I don't have any reason in particular to think that Pipewire would help, ... but sometimes changing random stuff solves the problem without needing to get into diagnosis.