We were asked that once (before my awakening), and my response was, "I'm not sure, let me check with my wife and get back to you."
I forgot to get back with the elder. He never said another thing and we didn't get the visit.
----
I'm definitely going to use that tactic when my time comes.