Chances are with his busy schedule (especially at the meeting) he may not even mention anything. I remember a mate of mine and his marriage was failing and an Elder asked him in front of the whole group if his wife was ok (she had not attended for months) he simply replied 'yes she's fine' , I think this approach threw the Elder off balance, I mean what do you say next to that reply unless your a kamikaze person who wants to out all the potencially sorded details in front of everyone?
Again using that approach, if he came up to you and mentioned about coming round you can simply reply 'yes I know I was in'. They only have power and authority over us because we give that to them, as soon as we stop doing that, it melts away.
CS 101