"Some weird reason" being that the Census Bureau hasn't engaged in cyber attacks such as Stuxnet

I think you are unexpectedly proving my point. The answer is bias.

I've actually known a few people that had a real problem with the census.

I don't know why you think them not being law enforcement changes anything?

Constitutional prohibitions generally apply to all governmental bodies, i.e. a school is limited in its ability to constrain your freedom of speech or religion.

The difference there is that the first amendment applies to actions of the individual but the fourth amendment applies to actions of the government. That difference in scope of application thus determines not just whom is protected but also to whom is limited. These first ten amendments exist explicitly to provide individuals lawful rights and when these rights are not violated these laws thus do not apply. For an example see the exclusionary rule. Thus the 4th amendment’s application does not apply all functions of government equally and no law or prior practice says otherwise.