The constitution and subsequent amendments merely outline which rights the government may violate as a matter of necessity to run, and certain rights which it may not violate at all.
As you say, privacy is not mentioned; it does exist as a right but has no explicit protection except for the conditions in the 4th and 5th.
Of course, the federal government's role has grown so much, and the interstate clause abuse has basically expanded it's power to encompass anything and everything, it is not unreasonable to think that the first 10 amendments are the only rights we have anymore :/
Scalia believed strongly in the right to privacy and privacy was certainly considered a right at the time of the constitution. I'm not sure where your source got that idea, but it's easy to document otherwise.
https://iapp.org/news/a/scalias-privacy-impact-will-be-felt-...
"A machine that sees through walls, reasoned Scalia, in order to capture the heat emanating from lamps used to grow marijuana, infringes upon individuals’ reasonable expectation of privacy, even when law enforcement is positioned outside the house."
http://caselaw.findlaw.com/us-supreme-court/533/27.html
"In United States v. Jones, probably the most important Fourth Amendment case since the 1967 decision in Katz v. United States, Scalia, writing for the majority, required the police to obtain a warrant prior to attaching a GPS tracker to a suspect’s car. While basing his opinion on narrow property grounds, Scalia rejected the government’s claim that individuals had no expectation of privacy while driving in broad daylight on city streets."
http://caselaw.findlaw.com/us-supreme-court/10-1259.html
"In Florida v. Jardines, Scalia, again writing a 5-4 decision supported by three liberal justices and Justice Thomas, held that police use of a trained detection dog to sniff for narcotics on the front porch of a private home was, indeed, a search within the meaning of the Fourth Amendment. The consequences of Jardines, as well as other dog sniffing cases, are far broader than for just narcotics searches. They implicate suspicion-less searches by other mechanized sniffers – for example, think about keyword searches of communications contents and metadata for cybersecurity or intelligence purposes."
Scalia believed in protection from government trespass on private property. That's a sort of "privacy" but its based on property rights, which was the original understanding of the fourth amendment.
> The way Justice Ginsburg saw it, Roe v. Wade was focused on the wrong argument — that restricting access to abortion violated a woman’s privacy. What she hoped for instead was a protection of the right to abortion on the basis that restricting it impeded gender equality, said Mary Hartnett, a law professor at Georgetown University who will be a co-writer on the only authorized biography of Justice Ginsburg.
Ginsberg realized that “equal protection” has the benefit of being in the constitution, while “privacy” nowhere appears in the document.[1] As the left has shifted away from social libertarianism, the ideological underpinning of Roe has become increasingly unsettled. That’s why in the recent confirmation hearings you only heard discussions of “super precedent” not any sort of debate about a sweeping privacy right.
[1] “Privacy” as an “unenumerated right” makes little sense under either liberal or conservative readings of the constitution. If you, like Scalia, believe that the “unenumerated rights” were those that were understood to exist at the time of the founding, then privacy isn’t one of those rights. But even if you subscribe to a “living constitution” view that fundamental rights can change over time—what’s the basis for your claim that “privacy” is generally recognized as a fundamental right today? I’m not aware of any constitutional court in the world that has embraced a notion of “privacy” broad enough to include both abortion and freedom from surveillance. Around the same time as Roe courts in Canada, France, Italy, and Germany declined to recognize any right to an abortion. The first three found it to be a legislative matter. The latter found legalizing abortion to be unconstitutional as a violation of the German Basic Law’s right to life.