If you are having a problem with the direct confrontation (and an inability to say no), I would suggest changing congregations if you can. They will try to talk you out of it, but do it. Say your family needs a change or something. They will send a letter of introduction to your new congregation.
No doubt the body at your new congregation will be happy to see another prospective elder on their body. This may give you a few months of elder-free duty for a few months until the CO comes around. He will visit your new congregation, they will probably discuss you briefly as a formality during the elders meeting for the week, and recommend you. They will send it off and when it comes back, probably within a couple of weeks after the CO leaves, they will ask you then if you will accept the recommendation. At this point...JUST SAY NO and that you would like to spend more time "building your family up spiritually."
It may be a little easier as you will have less of an emotional attachment to this new elder body. Believe me, I understand. In the future I will share more of my own story.
Axelspeed