May I make a suggestion that will actually help?
Next time "the elders" show up, tell them politely they came at an inconvenient time. Would they come another time and please phone first to make an appointment.
They will try and nail you down to a time NOW. Reply with "I'm sorry I can't give you a time right I have a lot on my plate right now, please phone next week.
Remind them that they wouldn't like you dropping in on them without prior announcement and that they owe you the same curtesy.
Tell them it would be best if they leave right know because a family member who is opposed to the society is about due to pop in for family related busines and it would be best if they weren't here when he came. He is the short tempered type.
When they ring a week later to make an appointment keep brushing them asside week by week. No explanation neccessary. Private family busines, you don't care to discuss it, thanks for calling, bye and hang up.
They'll soon give up.
Remember they are not the police. The only authority they have is what YOU give them. No more.
And you will know the truth and the REAL truth will set you free.
OlderTom