> vlc crashes under wayland and it takes the entire OS with it. I haven't discovered any smoking guns, but I assume it's some sort of OOM issue

As someone already pointed out, VLC doesn't support wayland natively, so it's running under Xwayland. Ironically the issue could very well be X related.