The amount needed would be huge and we could see the effects. This "dark dust" would dim/absorb the light emitted by stars behind it and turn it into infrared light, that we could see.

Also, since it is regular ("baryonic") dusk, the slightest distortion would make it clump together, create stars, turn into visible matter.

The last reason is the cosmic microwave background (CMB). We can infer from it how much baryonic mass in the universe should be and the huge amount of dark dusk does not fit in this calculation by a factor of roughly four.