I always have an unpublished phone number. I don't tolerate people in my life who give it out. I also let people know I don't accept "drop in" visits. When I get together with people it's by appointment only. I'm friendly to family and friends, but even they know better than to come by unannounced. When strangers or acquaintances speak to me I'm civil but not friendly. The door closes quickly if someone doesn't get to the point in the first 2 sentences.
Just politely say, "excuse me but I have to go to the toilet." Close the door and don't open it again. If they're stupid enough to wait that's their problem.
W