I don't use a DE, for much the same reasons you name, but I do use several pieces of LXDE, as they are generally built configurably and work well outside of the DE.