As a father of two and having exited the cult a couple years ago I say cut them loose. Life's short God dammit.
My wife and I have made it perfectly clear to my mother that if she decided to shun us if we ever got dfd that she'd be the one losing out, as shed be the one to cut her relationship with her grandchildren.
You don't deserve to be treated this way oh and I would tell your husbands ex to royally F@#k off!
Live your lives and be as happy as can be!