XFCE is a integrated desktop. You are choosing the desktop and what comes with it, including how the developers decide to change it. If you want control start with a window manager or a window manager like compositor for wayland and build your desktop from there. There are non-desktop versions of pretty much everything you need (notification daemons, keybinding, etc.).