If you were saying he should 'go back to where he came from' because you think it is simply better to send all criminals back to their country of origin - the guy deliberately tried to run people over in the street.
Can I just say 'go back to where you came from'? Do I have to explain my statement?
What if someone else called me racist for telling him to go back to where he came from? They won't believe me when I tried to explain that I don't hate his skin colour, I hate his actions. What then?
... because that's the situation we find ourselves in today, in 2019. That's why Min started this OP.