Perhaps Zid, I'm just basing this on me. When I was a believer, this was not a difficult concept for me. I could completely grasp that my brother's atheism was a lack of belief that was nothing like a religion. Maybe that was just me. Perhaps most really can't conceive of it, since the world for them is defined by their OWN beliefs, and therefore all must be processed through that filter. Considering how many things a believer in the judeo/christian god does not believe in (Thor, unicorns, elves) one would think they could grasp the concept more fully. I never define someone by their disbelief in Thor, claiming that disbelief is a religion unto itself.
It's all ridiculous---and quite ethnocentric---but that's how we are.
NC