For me the primary problem is the lack of reciprocal acceptence.
They want you to accept that you are not 'good' enough to be associated with, but they see your child as a potential convert and want you to allow them to associate with your kid, provided you are not there as you're not good enough to associate with.
If you allow this you accept their valuation of you as right and valid, and we all know that it isn't true.
If THEY decide that they cannot spend time with your child in your home, it is THEIR decison that stops them seeing the kids.
You have no obligation to, in effect, let people spit on you in front of your child and then let your child be taken off by the same people who spit on you.
This might be an extreme way to put it, but I really feel for your own self-respect and your kid's perception of your self-respect, you cannot allow this.
Either they accept you and your kid together, or they can have neither.