I read good advice from all so far. I want to echo the advice already given by problemaddict not to go into the specifics. do not let them drag you into a conversation about how you feel about about the organiztion and the FDS. That always comes up.
Say you are not there to comment about anyone's relationship with Jehovah or claims, that you can speak only about yourself. Say that you do not have a message that you think needs to be heard, that you have personal issues that you already talked to the elders about that prevent you from speaking before the congregation. And that you are content to let the matters rest. You have to be somewhat content to let the matters rest simply because you cannot change the organization by attacking it locally. You could add, as you already mentioned, that you really don't want to talk about it any more.
I was similar to you, an elder a long time with many privileges in the Circuit. When I stepped down the CO also talked to me. He did not dig too much. He was most concerned and looking to see what threat I was going to be to the congregation. Once I relieved him of that concern he was done and ready to go. Bob in NC