To "drift away" you still have to make some kind of statements here and there. You can say "I don't wish to discuss it." You can say "Thanks for your encouragement." You can give the impression that you will be back soon (and then don't be back soon) or give the impression that you will not be back soon.
I pushed back a bit more and said "I have doubts I need to work out." I scared the body of elders enough with that type of talk that they wanted to avoid me. That won't typically work for others. But you never know. Now, you can say how you read about the Australian Royal Commission or that you have to meditate upon overlapping generations or even how the Governing Body asks us to obey directions without questioning them. You can say you have been disturbed ever since "they" took congregation funds away yet ceased the worldwide building.
Many who fade away simply say "We are fine, we know how to reach you if we need to." That's a non-answer that indicates you probably won't be back soon.