This is the biggest hole in your argument:
If you go up to somebody in the street and start shouting and swearing at them and calling them obscene names you should expect to be arrested. That has nothing to do with limiting your freedom of speech.
Racists would (and have) said that they can do this as under freedom of speech. "Sticks and stones may break my bones but names will never hurt me" and all that nonsense.
They are of course utterly wrong. Calling people names doesn't kill them, but is does cause psychological harm and that is why we won't permit this.