to see how much the runtime contributes to the size of the binary use an empty main. for hello world you bring in the fmt package which is a relative heavyweight:
$ cat > t.go
package main
func main() {}
$ go build t.go
$ wc -c t
623280 t
$ nm t | wc -l
1029
$ cat > t.go
package main
import "fmt"
func main() { fmt.Println("hello world") }
$ go build t.go
$ nm t | wc -l
2541
Removing DWARF information with -ldflags -w reduces the size of the binary by 20%. The binary is 1.8MB originally...
the rest is pretty much all runtime, the hello world itself should take about 10k, that's more or less the size of the C or dynamically linked Rust versions (Rust also uses static linking by default, and the hello world is 300k in that case)