The good thing about Ubuntu is that it has so many forks: Xubuntu (Xfce), Kubuntu (KDE), Lubuntu (LXDE). They all use the same Ubuntu repositories. For example, you can get an Lubuntu system by removing the unity and gnome package from Ubuntu and installing the LXDE ones. I held off upgrading my Ubuntu machine from 10 because of the Unity thing. I installed Lubuntu 12 on another laptop last week and everything works great and there pretty much aren't any surprises going from 10.

What's annoying isn't Unity but the way Ubuntu changes the way startup scripts are done and small things like .xsession don't run by default (and if you turn them on, suddenly .Xmodmap stops working...). That's true for the Ubuntu derivatives too. But that's a fair trade-off for good drivers (even if they are proprietary) and up to date packages. Otherwise I'd go with Debian testing.