Agreed it is a hard balance, all gets down to how responsible the average person is and working around that aspect.

Universal free speech as it stands, allows racism and with that, you are not wrong. Free speech within some moral boundaries, though free debate, that is the aspect of free speech to provide balance that is equally important.