It is emotional and irritating when dealing with someone that is trying to friend or re-friend us when we know their friendship is only conditional. We know the JW system and how it controlled and burdened us as young ones. Your nervousness may be a first-step mechanism indicating the need of alert/protectionism. If they come again, hopefully if your husband is there, he can take it to a second-step and tell them whatever he wants, like they better not ever come over again, etc. My husband now, never a JW, told a couple male JWs (I don't know any in this area)when they came to our door, that he was really glad they came by now that he got df'd for apostasy and that he really wanted to tell them "the truth about the truth." He was quite surprised how spooked they were & left immediately.
Your children are very fortunate as they have a mom, now with a certain kind of experience, to raise them with freedom of mind and heart. You can ensure them of your unconditional love and provide a normalcy for them.
So many best wishes.