You can just install the command line tools without xcode. From those tools, most can be replaced with their homebrew versions so you only need them for bootstrapping. But if you want to do hardware- or OS X related development, you will need to keep the tools around. CUDA, for example, needs clang et. all.

They don't take much space, though, and the toolchain is treated a bit better by Apple than utilities like git, vim etc.

I don't know about homebrew, but macports has clang and other toolchain stuff, so I think it might be possible to uninstall Xcode after installing all necessary tools through macports?