I always somehow feel crummy when I say I'm an atheist. I shouldn't, but it seems to imply lack of heart or something.
Yes, the world (especially American culture) places unfair connotations on atheists. You're a God-hater, or immoral, or what are you doing celebrating Christmas?
That's why many choose words like agnostic (which just means you don't know) or unbeliever, which is closer.
I found that by joining my local atheist group, and hearing the very interesting thought process and conversations, and feeling their humanity, that I'm slowly overcoming my shyness about the "atheist" term. I think if more non-believers adopted that term, then more believers would see that atheists aren't really that bad after all.