What about the future when it comes to birthdays and holidays like Christmas? If you would feel inhibited then they would still be a negative factor in your lives. So DA.
If you are leery of them in terms of potential slander issues, this being a small town, re your business or jobs. Then I would definitely go to an attorney and prepare a brief letter as outlined on page one (actual letter halfway down page) and set the tone by having them deal only with your attorney who will brush them off.
If approached by any of the elders refer them to your attorney. That on advice from your attorney you are not permitted to speak to them in their capacity as officials of their congregation in case any civil issue arises. Tell them you are not speaking to them out of love.....LOL.