It is stressful because it shows a complete disrespect for your thoughts, feelings and expressed wishes. It means that they can and will visit unannounced and that they are in control of when they turn up at your house and not you. It is basically an effort to reassert their authority however clumsy.
However that being said your home is your home and if they turn up announced then state that further visits are unwelcome unless they are expressly invited since they have not respected your wishes. You are aware of their concern and know where they are so will contact them if you have need of their assistance and encouragement. Then say goodbye and shut the door. How polite you choose to be will depend on you and your circumstances. They only have the power over them we choose to give them.