What is hard is if you still have family in.
I was an elder for years, had parts on assemblies, served in India for a while and had a model family. But then it all fell apart overnight.
However, my wife still attends and is a very faithful and loyal Witness. I have always respected the beliefs of others, even when I was active so i am not going to pressurize her. It is hard sometime when you personally would like to move on, I have different views now but I don't want to anything that would get me disfellowshipped, even 5 years down the line from giving up attending.