The question of his motive is crucial. If - and I think it's a huge "IF" - there was an affair, then it will slip into obscurity. Most uninvolved readers would think, just another straying church member reaping the wages of sin.
On the other hand, if his motive was more obscure - or bizarre...
It will be interesting to see how this develops.