That is one seriously disloyal wife you have, going and telling those men about your private lawful conduct you do in your own home.
It's none of their business and you are under no obligation to talk with them. And her, I find that conduct untrustworthy.