Alot of good advice from the posters here. Your story does not add up to me at all, you were having an affair with him, yet sharing a basement suite with him, then he would be with his wife. I smell something that stinks alright. This story, its as discombobulated as it gets.
If you were having an affair with the guy, no sympathy from me regarding how you have been treated. I've been the wife who's been cheated on..it sucks! However, its also true that he was the married one and is the true SKANK. Still to go and sit by his wife and get a kick out of it is very immature and selfish. What about this child? I am assuming you don't have children of your own, life has a funny way you know. What goes around comes around and someday you may know how this wife felt.
That being said...you dont' want to be a witness, so why go to the elders? To punish him? To destroy this child's family? To payback his wife for her comments? To payback the people who now ignore you?
It's pointless, because all the looks and comments, you have brought upon yourself by your actions. Period.
If it is revenge you seek, stop being so selfish and think of this innocent kid. It will be the first step towards maturity you take. Good luck.