Because, when provisioning a new server, where else can you get it from?

A host that you control.

Where did it get it from?

From github, the first time only. Not every single time.

Isn't getting it from GitHub the first time vulnerable to attack? Consider when you're scaling up or replacing a failed host.

From a custom package you have built? Did you know that RPMs and DEBs don't grow on magical trees, but are built?

So, the problem still stands, theoretically.

Not only do you likely need to populate your host with packages not from your host. But also, your host will also still be connect to a public net, even if only indirectly (e.g. private net), and hence potentially manipulated.

No, you misunderstood what is actually the problem here. Pulling under deployment some code from random resource from the internets that can go down or get deleted at a whim and you can't easily move to just some other mirror and you don't even control when the thing will be up back, that's the problem. Not the trust you need to put to use the code (this is still there, obviously). And the very same comment applies to third-party package repositories, like PPAs in Ubuntu.

Not to mention that with pre-built binary packages your deployment speed and repeatability get significantly better, as you don't need to rebuild the artifacts every single time.

I do understand, but "the trust you need to put to use the code" is what we were talking about, no?