Honestly sweetheart, I'd say cut your loses and go. Chances are his wife has suspicions. Chances also are if you told other elders in his hall, he'd say you were lying OR he'd pull the repentant preacher act.
You've got nothing to win here. Take your freedom & don't hurt yourself any more by having anything further to do with this arse.