I agree with Comatose. I had responsibilities like yours. I know people wondered (and still do) about me. But I was treated the same. In fact, I still had others coming to me with problems, Bible & organizational questions, etc. I still had elders coming to me for help with parts, discussing problems with other elders, etc.
I think one thing is that I never acted weak or apologetic. To put it bluntly, I don't take much junk. They know me and my abilities. If you step aside, tell them you want to become more spiritual. Don't say it sheepishly as if you're weak. Do not say it apologetically. Say it strongly as if it's a well-thought-out decision.
And I agree with your two friends; it has been absolutely wonderful to not be in the grind.
If your heart is in the right place, you might have some moments of missing certain parts of it, but you won't miss it much. In fact, I realize that a large percentage of my time was purely wasted as an elder. For example, I could spend a lot of time just trying to get a certain talk to fit its alotted amount of time. That wasn't spiritual. I wasn't learning anything. A lot of my study time was just preparing parts - not learning and not really questioning and researching. A lot of it was just orchestrating interviews and all kinds of stuff like that that I feel was a total waste. I would ride two or three or hours out of time for a pre-circuit assembly or pre-district convention meeting only to get there and find out it was a waste of time.
Since you are getting older, use your time wisely. If Jah is who you think he is, he will understand you and appreciate you and you will be rewarded. Search for truth and humbly (& strongly) be willing to go where your search leads you. Study the Bible deeply. Consider different translations. Research the canon of scripture used by JWs and others. Consider why other books were not included, who determined the present canon and whether they can/should be trusted.
I actully feel nobler and stronger now. I know that my study and seeking now is genuine - not for glory. I want to find real answers - because I hate suffering and injustice, I love life, and I love the planet and am fascinated by the entire cosmos.
If there truly is a loving god, and if you are a strong person who loves justice and kindness and hates to see others suffer, and if you are willing to put forth effort, then I think that god will be pleased with you and you will fare well. If there is going to be a new world as JWs teach, and if you are as just described, I think you will be a prince in that new world. At that point, nobody will even remember who were elders in this world. Nobody will remember who gave district convention talks in 2014.
There is something wrong in JWdom right now. If you still believe in the JW way, then maybe you could view it as being like the times in Israel when there was much corruption and only a few noble, righteous souls. Finally, Jah performed cleansings. Maybe there's going to be a major cleansing among JWs.
So, step aside. Start engaging in real study, meditation, and prayer. Ask Jah to help you unbiasedly and open-mindedly find and see truth. Whether you realize it, you've actually been sort of kept in ignorance as a JW. JWs don't do real, in-depth study. Your time has been eaten up by busy work (probably intentionally by JWs).
Watch to see what's going to happen in the JW realm. It's interesting. Maybe future happenings, changes, etc. will help you to see more clearly where you stand and help you to sort out feelings.