Theories like IIT are just barely one step away from magic IMO. Consciousness is an intricate, macroscopic phenomenon, so to require it be present in every microscopic spec of matter just because we can't get a grip on qualia seems like a serious overreach.
If experience is irreducible, then something else is. In physics, that would probably be fields. Irreducible doesn't mean magic. It means we can't find anything to reduce it to.