The default Ubuntu PPA did not work for me. Then I found this custom PPA with newer versions: https://launchpad.net/~pipewire-debian/+archive/ubuntu/pipew...
It took a bit of hacking but it was most likely the mess I created earlier (half-way through I realized I had some leftovers from when I tried to build it manually some time ago).
In the end I managed to remove all traces of PipeWire from my system and followed some tutorial which referenced above PPA, without any further problems.
Looks like it is super easy with Nix but I've not yet had success on Ubuntu.