One confounding factor is backends. On one hand, GCC has a lot more of them, which contributes to a larger source base, but shouldn't be considered a disadvantage out of hand. On the other hand, LLVM builds all 13 of its backends into the binaries and libraries by default (compared to GCC's single-backend approach), which contributes to larger installed files but shouldn't be considered a disadvantage out of hand.
As a minor point of order, neither LLVM nor GCC include a linker; they both shell out to the system linker (often GNU ld from binutils). LLVM now has the equivalent of an assembler (direct object code emission) for those backends that support it, and the nascent lld ( https://github.com/chapuni/lld (warning: i couldn't find the webpage in www/ hosted anywhere, this is as close to something canonical as I could find) ) looks to be the beginnings of an LLVM-affiliated linker.
LLVM + Clang source code: 9.9MB + 6MB = C, C++ and Obj-C compiler and linker
GCC source = 69MB = just C compiler and linker
G++ source = 6.6MB = C++ additions to GCC
So I'm curious as to where you're getting the "hundreds or thousands of MBytes on my mac" from.