Hello world is large primarily because it statically links in jemalloc and libunwind, which are obviously fixed overhead. Both can be removed (and indeed I would expect jemalloc to be turned off by default sometime soon because each jemalloc release seems to be less stable than the last). If you do this you'll get a much healthier binary size.

C binaries are small because they're assuming you've installed what is ultimately the C runtime on your system (libc).