You're right, it's an implicit assumption in the declaration, made explicit in the Constitution.

> We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness

Rests on the assumption that slaves are not "men". But something else.