That's why there was a great initial push after the Constitution was ratified to establish the Bill of Rights. People didn't feel that it was safe enough to leave things like the freedom of speech to be covered under common/unwritten law. So some of the unwritten/understood parts of the Constitution were written into the first 10 amendments.

There was also great initial resistance, not because the opponents didn't want to safeguard rights; but because they feared enumerating some rights for named protection would leave other rights not named unprotected.

And if Hamilton could print out this discussion page, he'd be shoving it in people's faces and pounding his fist on the table furiously. "See? I told you so!"