The president and congress are just as sworn to upheld it, and with respect to several portions of it (e.g. impeachment) are the only arbiters of what it means. With the rare exception of cases where the Supreme Court has original jurisdiction [1] - congress is even allowed to replace the Supreme Court with a new final court of appeals. The Supreme Court has surprisingly little constitutional power if Congress decides to sideline them.
[1] > In all Cases affecting Ambassadors, other public Ministers and Consuls, and those in which a State shall be Party, the supreme Court shall have original Jurisdiction
[2] (Emphasis added) > In all the other Cases before mentioned, the supreme Court shall have appellate Jurisdiction, both as to Law and Fact, with such Exceptions, and under such Regulations as the Congress shall make.
And again, in theory Congress can make exceptions to it, but in the end its still up to the Supreme Court to decide that they make sense. A piece of paper has no meaning in and of itself.