Considering that the celebration itself is a mix of cultures and religions, I don't see a problem at all. Case in point, I can decorate my house, give gifts, invite people for a feast, and everyone sees it as Christmas.
If you switch things around, would Christmas be fun if you removed non-Christian elements from it? Such as the tree, Santa Claus, and the elves?
In the end, why do people celebrate it? Some believe that, at least, in its origin, it really was to celebrate the birth of Jesus. But if that was the case, there wouldn't have been a need to add all the other elements from existing "pagan" celebrations such as: yule logs, evergreen trees and winter solstice celebration. We may also consider St Nicolas, who what added centuries later! Sure he was a generous Christian father, but that was not enough. They had to give him non-Christian magical powers, elfs, and flying reindeers!
Still, all these additions were accepted by the majority until the celebration itself became something else.
In the end, what people really care about is not really the event that they are supposed to commemorate, but the reason why the celebration existed even before Jesus came along: It is to spend time together, to celebrate at a time where there is less light than any other time in the year, to have a reason to travel and see friends, family and perhaps, spend several days together.
And this is exactly how atheists see it: It's a holiday for fun, family, and friends.