I remember running into an inactive witness about 20 years ago (I was still active then) and saying to him, "I haven't seen you in a while." He looked at me and said, "That's because I haven't been anywhere for you to see me!" I was so taken back because his comment seemed so harsh - i was geniunely surprised by his response.
Now, I'm the one who hasn't been seen in a while... and when I run into them, I smile - and greet them. No one has put me in the position to have to answer any questions. If it comes up, I plan on politely saying "I'm not attending anymore."