The first amendment is a legal principle with clear implications, limits, and enforcement mechanisms. This makes it much easier to discuss rather than the more nebulous “free speech” which allows everyone to just make up their own definition in-situ, leaving the conversation unmoored from any common definitions.

Aside from that, removing the first amendment from my argument changes very little. You’d still be obliged to create a definition of what is and is not free speech and a mechanism for enforcement, which would be subject to the pitfalls listed above.