Me and my wife kind of did the fast fade. I was really sick and stopped going to meetings while she went a couple of times and when I was better I told my wife, I'm not going back and she said, ok. We got a visit from a couple of elders once but we were HBH and they have bothered us less and less over the years. The thing that hurts though is My wife's sister is only 15 and her mom limits their association even though they are really close. But considering some of the sories I've read, I'd consider us lucky.
I'd just want to encourage you guys that you are doing the right thing for your immediate family. Your witness family may shun you, but hopefully some of them are less rigid.