As Rocky Girl said, they only have as much authority over you as you allow. You've already said you don't want to discuss it with them. So if they approach you, just keep saying that even though you don't owe them any answer at all.
If a police officer pulls you over, you are required to show him whatever documentation he requests (license, car registration, etc). If you receive a supeona for a court appearance, you are required to go and to answer the judge's questions.
However, the elders are not in such positions of authority. They are not police officers, they are not court judges. They have no more authority over you than I do. They have seen that you have not been at meetings. Now they know (thanks to this tattle-tale false friend) is it not because you have been in the hospital or overseas or are dead; they can conclude the reason you haven't been there is simply because you do not want to be. They don't need a college degree to figure that out. Nothing else is owed to them.