I'm fine with the idea that it is hypocritical to celebrate Christmas while not believing in god. My comment about Labor Day was facetious, but could be seen in the same light. I celebrate Halloween, though I do not believe in demons or witches or ghosts. I celebrate Thanksgiving, but do not say a prayer of thanks. There are likely to be other such celebrations that I partake in without believing in the subject being celebrated.
I doubt most people would express any concern over it. It sounds similar to saying that I'm a coward because I refuse to drive a large knife through my hand. It's true! I refuse to do it, and it doesn't bother me to be considered a coward in this context. I am comfortable making that trade-off, and many like it.