No offense but it seems to me like "Well the frying pan was pretty hot so let me see what the fire feels like."
I believe it's the organisation of worship that hurts people, not the act. It's sometimes difficult to believe in something if there isn't a group of people to share in it and make us feel part of something. But when people get involved, peoples opinions get involved.
Just my opinion.