There are other more complicated reasons but this is a pretty good answer. Basically that matter at some point must have been at the Big Bang and with all the heat generated around that time some of it must have interacted with the debris that GP is talking about. We don’t see evidence of that. It’s important to remember that though we as humans lose things all the time. The universe cannot. We got what we got.