Be too busy to answer the phone or door. If it is too late for that, simply put off any meeting (work, travel, under the weather, and offer to meet at an impossible time..for everyone (late, early, etc).
Ask; WHY do you want to meet? WHO said what !?
Smile, chuckle, and say: I think she/he misheard me. They are confused. etc. Do you have a 2cd witness to confirm this rumor ??
Answer all questions with a question...like Jesus did. Why do you ask? What a thing to think! How is YOUR family, kids, work, health, relatives, etc?