Flag I did the same thing. The last meeting I went to I actually gave a talk. I was always one that was considered to be exemplar.
And the rumors started flying about 2 weeks later. Then elders started calling. I dodge allot of calls becasue I didn't want to talk to them and at the time I was afraid of stubbling them. Not for the sake of man but for what I had been conditioned to believe was for god. Anyway, after a while people just started leaving me alone. I decided to briefly say something to the 2 family members that were still in (my mom and sister). And I decided to say something to just a couple of friends.
But for everyone else that eventually just quit pursuing me.
With time everything blos over.