It has been said in this thread but I want to emphasize it.
This goes against all the training the elders get of never contacting a person from the opposite sex alone. Specially when the spouse is not a JW. You must talk with your wife first about the issue, and make sure she understands that is not because you hate the Jw's. It is to avoid reproach to Jehovah's name (use the JW lingo) then talk with the Coordinator of the body of elders and have this guy present. Let them know that if he does not STOP this asap, you will contact the Circuit Overseer.