So what if no malice is intended? It is the consequences to others that are important. Political correctness isn't aboutyou...it is about the ones who are marginalized/silenced by dominant culture.
The intention may be good, but the reality is that life isn't fair. I've been on the wrong side of that unfairness as much as I've been on the right side of it. Constantly altering our language so as not to offend those who feel discriminated against actually only serves to highlight their differences and eventually smacks of patronisation.
Refraining from obvious insults is just plain good manners. However, insisting that the dominant culture walks around on eggshells for fear of causing offence - such as saying "Merry Christmas" will only result in many of that dominant culture becoming wearied and unempathetic to the minor culture.
It's all a question of degree and intent IMHO. After all, being offended hasn't yet proved fatal to anyone.