As someone who recently installed GNOME on a Xubuntu system and then uninstalled it, I can say firsthand that that still leaves a bunch of cruft, does not cleanly remove all packages.

It's not GNOME's fault though -- it's an annoying issue with big metapackages. Same thing happened with XFCE on a stock Ubuntu install.