I think it depends on the hall you decide to move in. Some halls have lazy elders who could care less that you left, so they won't bother you. Some halls have Nazi Elders.
We had lazy Elders, so for the most part they left us alone. Our problem became that we could not stomach the hypocrisy anymore so just started missing meetings and assemblies. It took us about a year to stop reporting time. Wow...life if now good and it is great to make new ("worldly") friends that like you for who you are and don't judge.