There was generally always considered to be 5 things that were not protected under the First Amendment. I doubt think any of the founders would disagree with the notion that the following where not protected under the First Amendment.

1. Libel

2. Slander

3. Treason

4. Certain threats / incitement to violence (like hiring a hit man)

5. Obscenity

All of those are examples of limiting what is considered free speech, none of them limit who is allowed free speech.

This is a simplification that is not true. The U.S. limits speech via trade secret laws, copyright laws, libel laws, and laws regarding threats.

There is no limit in the text but it has been limited by the Supreme Court. Supreme Court has the force of law.

I could only find one ruling limiting who is allowed free speech, though my search wasn't thorough. Border control was allowed to bar people from entry based on protected speech made outside the country.