You're not under obligation to do ANYTHING. Do what you want to do!!
These people have NO power....only the power YOU give them.
I will say your observations are true: some are able to leave and be left alone; others are hunted. I think it comes down to the personality of the elder body.
I was a well known elder and got up and walked out one day. Not one soul has bothered me. But....to be honest......I make it known that if they come anywhere near me, I'll treat them as a threat..... and they are rightfully scared.
Do what you want......you do NOT have to write them a damn thing!