I am SO glad that my hubby has never been an MS or an Elder! My x-SIL HATED it when her hubby was an MS. She had 2 small kids and was left alone with them at meetings AND especially at the assemblies. She finally asked him if he would step down so he could help her out.
I agree the Family aspect! Be there for your kids and your wife! As you said......Concentrate on spending time with your kids........They will only be with you for a short time, and you will never hear them say they wished you had been an elder and spent LESS time with them! THEY need you NOW!
My kids are 13 and almost 17. They NEED their dad to be here for THEM, not off counting contributions or dealing with other people's family issues.
I am proud of you in that you recognize the more important things in life.