It will work under wayland if you have xdg-desktop-portal/xdg-desktop-portal-wlr set up.
If you've got that set up up correctly, screen sharing will also work in Firefox (for instance on discord).
As far as I understand it, xdg-desktop-portal is a DE/WM agnostic protocol that enables applications to easily capture a screen - the user just has to run the right backend for their environment. I think it does other stuff too, but screen recording is probably the main use case.
I'm using Manjaro Sway Edition where that was configured out of the box.
I log in into X11 when I use OBS on Ubuntu.