Because the 13th Amendment doesn't prevent Congress from making a law to make people slaves, it prohibits slavery from existing.
In law, words make a difference.
> Why can you sign a contract that limits your freedom of speech in spite of the 1st Amendment?
Because restricting such a construct -- preventing you from trading speech or restraint of speech for some other good -- would violate the prohibition on government regulation of speech applied by the First Amendment.
Also, I will note that you point out that the specific words are important in your first point, but miss it your second. You mention that the First Amendment is about not allowing the government to regulate speech. But the actual text says that Congress is not allowed to "abridge" freedom of speech. Abridge means to limit, shorten, or cut off. Clearly, a law that invalidated contractual limits on speech would not in any way be abridging that speech.
I agree, though, that there's a Shelly v. Kraemer type argument that could be made against the enforceability of contracts limiting freedom of speech in certain ways.
Why can you sign a contract that limits your freedom of speech in spite of the 1st Amendment?
Yes, I realize the law is complicated and there are many years of precedent distinguishing the two. But it's incorrect to argue that this isn't even in spirit related to the 1st Amendment.