Thanks for all your answers. I have been asked this question by an elder a few years back. There was only one elder, so I knew they didn't have their two witnesses. I also knew it was the question they used to try to claim that someone disassociated themselves.
Because of only having one elder there, I knew I had some wiggle room. I still wanted to be left alone, so I turned the question back on him. I acted shocked and asked if he asked because HE didn't believe it was true. He recoiled from the thought and that was the end of the conversation.
I am glad I had made the rule that I would only talk to one person at a time. I refused any visits by two elders. Now, I just refuse any visits. They still try to catch me working in yard and will use that as an excuse to stop by and talk. I will talk to them about everyday things, but that's it. My own thoughts on God and religion are my private thoughts and I choose who I wish to speak to about them and when.