Getting there with Ubuntu was pretty much "apt-get install" when I made the switch to i3wm. Sure, you end up carrying along hundreds of MB of "junk", but I just didn't find it worth the hassle to trim it down - I have far more junk than that in form of data of my own I could delete...
What you're describing was pretty much my Archlinux setup btw. It was just more convinient for me to build it from an archlinux than a regular ubuntu. You could with an ubuntu though, it'd just be a lot more work I feel like.
EDIT: turns out I'm pretty much explaining you what you already know.... I'll leave my comment up anyway, maybe it'll help someone else.
> My personal opinion is that i3 rocks, and and that I don't need all of Ubuntu or Xubuntu