He said you can have privacy without secrets. But suppose there exists a room with a door where you don't know what someone is doing behind it. The activity behind the door is the secret. The door is still representing privacy (not secrecy). That was the point of the analogy.
A secret is an attribution to the contents of a description which are withheld for security reasons. Or at least, that's what it should be. And that's what it appears to be with the military.
Privacy is a right to withhold information based on a fair desire to not have that information be known.
Invasion of privacy is someone pulling the curtains when you're showering or opening the door when you're in the bathroom.
That should not be confused with invasion of secrecy.
Either way, the main point is that these two terms are confusing, and even the law confuses them. And yes, people would consider something they don't know a secret also, and your use of the word is intuitively correct. But a clearer distinction could and should exist, and we should all work towards that clearer distinction. I believe the words are already more than adequate, since we already see the correct (most practical and fitting) distinctions used where they are most needed in practice (military).