At the same time: freedom of speech goes about as far as what it causes; you are free to say what you want, but nobody has to give you a soap box or audience. That is the same thing that is basically happening in NZ. If you think the colour of your skin is better than the colour of someone else's skin, that's your issue and the internet/press/media etc aren't going to provide you with a platform. And if you try to do it anyway, you will get a reaction. (i.e. jail)
In the US, even with your constitution, you have similar situations. If you want to start a clan of people who dislike other people and try to have club meeting and put those in the newspaper or on TV, you will also get a reaction (jail, fines etc.).
You say that one is "free to say what you want", but then immediately describe a case of a person literally saying what's on their mind and getting jailed for that. You're welcome to argue in favor of such censorship, but whether it does or does not have a valid reason, it's still censorship - and so not free by definition.
Oh, and the scenario that you describe is impossible in US. It is perfectly legal to start a private club that discriminates in membership, even against otherwise protected classes, and to advertise it as such. It only becomes illegal when you offer a public service.
You've been living in the US for more than a decade, so I don't think your "even me, a non-American, agrees with him!" card is as strong as you think.
You are right about that. 99% of countries on Earth have weaker constitutional rights than the US in terms of Free Speech - if any right at all.