Oh yeah. I fit in at the congregation I grew up in because I had roots there. I moved not far away and started over and tried for years without ever fitting in. My wife and I tried hard, even had pretty much the entire congregation over for chili parties in small groups one year, but still nothing. We would go out to eat after meetings or just in general and come across the same cliques sitting together at a restaurant while we sat alone, over and over again. We heard tales of their epic camping trips that we were never invited to, how they all went up to watch a football game that we weren't invited to, and so on. I once bought a bunch of pizza because I had a bunch of young people that were supposed to come over back when I was single. Not one single person actually showed, they all went and did something else. On the plus side, I had leftover pizza that I could have eaten for weeks. :(
For an organization that claims to be identified by love, they are the most cliquish, judgmental, small minded douchebags I've ever had the displeasure of meeting. There was no real love or bond, just busy work and meetings where people talked because they were there for a common cause. Once the meeting was over, so was everything else unless you were in the cool club and lucky enough to have your own clique. There was no real inclusion. Large families dominated the landscape and were the focal point of those that had any hope of being in a group.
Like you I had people turn away when I approached at times, and it hurts, but what hurts more is being stuck in a cult because you fooled yourself into thinking those people actually care about you. I have friends now that care about ME, not me with some arbitrary rules system in between us.