Whether it is selfish or not is not the question. We need to be selfish to survive in this life.
Life is to be lived...fully. However that sorts out, it is your choice. Live without regrets.
We all have those ties to the organization...either with family or our history. The importance of all things JW begins to fade more and more as we let go of it more and more.
Go live your life and have fun.