Not all artifacts are religious in nature, neither are they always assumed to be such. I think no one would assume a bust of Cicero or some random mosaic is a religious artifact.

But I sort of agree with the feeling, because it seems like it is that explanation which people fall back to in the absence of any other clue.