Fader, turn your weaknesses into strengths. You actually have an opportunity to step down (or up as Focus reminds us), without consequence.
How do I mean? Well it appears the body doesn't want you serving anyways. So give them what they want. Let them know fairly uncerimoniously, that you would like to step down for personal reasons. When the questions start, you can assure them its personal and that if you need spiritual refreshment you can get with them and blah blah blah.
If they really push, or the CO gets involved, then stick to the personal issues thing. Ask to be "respected", and if it really comes up, say that you can't serve with men you don't deep down trust anyways for the sake of unity on the body.
TRUST - is a buzzword for the CO's. You have to serve with TRUST. If you don't have it, don't serve.
Don't sell your wife short. Plan how to bring things up. DO NOT just burn it all down or assume other peoples reactions. Its the classic mistake people make. They push people away in anticipation of wht they know their reaction is going to be. While most of the time we are right, we can also be guilty of doing that premptively. So its important to at least give people the opportunity to react or chose properly. Most of all your wife.
My wife is a third gen, pioneer since she was 12, groomed to be a gilead grad, daughter of a legacy family who still assists in high level things like media coverage for foreign branches. A few CO's in the fam, etc... Yet she is about 80% out, and getting further along every day. I love my friends, and many of them will never feel the way that I do, but I don't care. Until they stop acting like friends, I will give them the benefot of the doubt. This approach has helped me maintain friendships I wouldn't have thought I could.