Hi!
Don't feel you are under obligation to explain yourself to everyone who are not entitled to pry into your business - take advice from the Insight Book:
*** it-2 p. 244 Lie ***
"Lying generally involves saying something false to a person who is entitled to know the truth and doing so with the intent to deceive or to injure him or another person."
Basis this you can, with a clean conscience, plan your exit strategy and misdirect those who are trying to interfere but have no business in interfering.
If you are young and still in academia then why not try moving. The change of circumstances and environment will make everything so much more easy.
Splash