Because coming up with criteria for determining what is and isn't conscious, and justifying the particular criterion you choose (as opposed to all the alternatives) is hard. Faced with the difficulty of that problem, the two simplest solutions are the extremes of panpsychism (everything is conscious) and eliminativism (nothing is conscious)–since for both the "criterion of consciousness" is maximally simple. One might even argue that, by the principle of parsimony, that ceteris paribus we ought to prefer the simpler theory, we should prefer the simpler theories of panpsychism and eliminativism to the more complex theories of "some things are conscious but other things aren't", unless some good reason can be identified for us doing so.
> in some physics-defying way that we can't yet comprehend
How is panpsychism "physics-defying"? Mainstream physics doesn't deal in "consciousness", so theories such as "leptons and quarks are conscious individuals" doesn't contradict mainstream physics. Well, it quite possibly would contradict the von Neumann-Wigner interpretation of QM, but I don't think many would consider that "mainstream"
Consciousness is ill-defined, also.
It does however seem to stop when the brain is destroyed, which makes it unlikely to be present in fundamental particles, or in parts of a disintegrated brain.
Does consciousness really "seem to stop" when the brain is destroyed? We can't directly observe anyone's consciousness other than our own. It is true that, when people die, we cease to have access to the outward signs we use to infer they are conscious – but we don't know whether that's because their consciousness has ceased, or whether that is because it has "gone somewhere else" to which we don't currently have access. We never have direct access to the inner reality of consciousness for anyone other than ourselves, and we can only know the temporary cessation of our own consciousness, its permanent cessation (if that is possible) is in principle unknowable to us.
Speaking of our consciousness "going somewhere else" at death (an afterlife) doesn't have to be a religious claim – there are heaps of "naturalistic" ways in which consciousness might survive death (e.g. Boltzmann brains, computer simulations, quantum immortality). Since we don't know whether any of those ways actually happen, we don't know whether or not consciousness actually stops at death.
Because consciousness is fundamental in a way none of those things in the world are. It is only through our own consciousness that we can know those other things exist, indeed that we can ever know anything at all. Treating consciousness as just a thing among things ignores that it is essential, foundational, to all other things as we know them, as we can know them.
I guess an obvious objection is "why?". Then something about Russell's teapot. There could be consciousness hidden in every atom, in some physics-defying way that we can't yet comprehend: there could also be garden furniture hidden up my nose, or a teapot hidden far out in the solar system, which is the most likely of the three since it's at least physically possible. Why think any of these things?
https://en.wikipedia.org/wiki/Russell%27s_teapot