They have as much power over you as you choose to give them.
To avoid a shepherding call it is good to keep it short, direct and non threatening.
Thank them for their concern, as you can see I am very busy right now and it is very diffivcult to find me in. I will let you know when things settle down and arrange a visit then. Thanks again I appreciate your efforts even though now is not a good time. Sorry have to go thanks for calling.
Keep in control, keep it short don't get into a debate or allow them to question you. Keep repeating sorry I have an appointment must go.
Really if you don't want to talk to them you are under no obligation to do so.