In order to claim a privilege you have to assert it first. So any time anyone sends something to the elders, you need to state in bold letters PERSONAL AND CONFIDENTIAL - CLERGY PRIVILEGE and then I would add further, not to be reproduced in any form and/or fashion, whether by copy or by verbal reproduction without the prior written consent of author.
If people want to keep things quiet, you should look up your state and its laws regarding clergy privilege and then identify anything you send to the elders as such. Again, you cannot claim a privilege unless you assert it first.