Source is often larger and you need to keep around the base source to figure out how to apply the diff and get a correct result.
What they're doing here is superior to source-based distribution if you care about update size and ease.
You would have to make sure that all your users have a (working) version of a compiler that is compatible with the libraries. For C++, this can be a royal pain. Even on Linux, where a compiler can reasonably be expected to exist, there's no way of knowing whether the system's compiler is what you need. And don't even think about Windows, where only developers have a compiler, unless you'd like to ship GCC/mingw with your original app...
There used to be an initiative called ANDF (http://en.wikipedia.org/wiki/Architecture_Neutral_Distributi...) which was designed to distribute software in some form of intermediate level code. They got compiled to native code on the user's machine.
I guess you could think of LLVM object files as the equivalent today.
I wonder why noone seems to have thought of it.