I'd rather spend my time treating them they way I wanted to be treated
That is why I wouldn't agree to a shepherding visit for myself. I would respect someone's decision to be left alone and to walk away on their terms. The JWs cannot, as they see their efforts as lifesaving attempts. But I am not required to humor them, especially when I know what they will do once they know I have no interest of returning. I can, however, leave them alone and I expect the same courtesy in return. If they cannot manage that, I feel no obligation towards them and would not feel guilty if I said no thank you, I do not wish to meet, you may take whatever action you deem necessary.
But I also respect each person's desire to handle such a stressful situation in a manner that suits them. Best of luck with the visit, I hope it goes as you wish for it to.