OBS doesn't work for me on Wayland either.
I log in into X11 when I use OBS on Ubuntu.
It will work under wayland if you have xdg-desktop-portal/xdg-desktop-portal-wlr set up.
If you've got that set up up correctly, screen sharing will also work in Firefox (for instance on discord).
As far as I understand it, xdg-desktop-portal is a DE/WM agnostic protocol that enables applications to easily capture a screen - the user just has to run the right backend for their environment. I think it does other stuff too, but screen recording is probably the main use case.
I'm using Manjaro Sway Edition where that was configured out of the box.
In fact the issue was closed without understanding what the reporter was asking: https://github.com/obsproject/obs-studio/issues/8822