I also think that if I were to say that to the four workmates -- even people I don't like and who don't like me -- there's no doubt that it would be coming from a racially motivated place.
That's why I would't ever say anything like that - a lot of people wouldn't say that because they have a filter, they realise that certain things are sensitive or could be taken in a certain way.
Telling people to go back to where they came is definitely said by actual racist people, but the sentence 'why don't you go back to where you came from ...' is ambiguous.
If Trump had been having a war of words with John Oliver and he told him the same, I don't think there'd be this much fuss and accusations of racism.
Again, the evidence that supports Trump's supposed racism is ambiguous at best.