Wayland doesn't have the equivalent of XGetImage for various reasons, but screen capture applications can use the ScreenCast flatpak portal to select window sources: https://flatpak.github.io/xdg-desktop-portal/portal-docs.htm...

I am not sure what you mean by Wayland can only be used sanely with toolkits, any GUI usually need a toolkit or some equivalent, even under X. if you are not using a pre-existing toolkit and are writing your own routines to draw buttons and text boxes and such, that would be implementing your own toolkit.