It's not just the 4th that alludes to privacy. The 3rd (quartering soldiers) infers a privacy of home, the 5th could be construed as privacy of personal information (specifically information that could be incriminating).

There's a ton more information here: http://law2.umkc.edu/faculty/projects/ftrials/conlaw/rightof...