Sure its easy.... the CO has limited time to squeeze visits in. He does it really as filler so he dosent have to do a lot of door to door crap.
Your busy. Whatever day whatever time they try to set up, you have something going on. Hasve a few handy excuses ready. Work doctor whatever... but your busy. Catch him next go round.
See? Your overthinking it a bit, really. Its not that deep. I doubt anyone has you pegged as an apostate. They noticed you slowing down and figured they would try to motivate you to full cultyness. Duck and weave and this wont be an issue