I know in my area apostate alerts travel fast, so you could be right about them being annoyed.
I find it interesting how all it takes is just a simple question or even a greeting can set them off, and you get to meet the real them - the one with teeth, ready to commit violence because you are the enemy "apostate."
If I see one in the supermarket now, they act like I should be apologizing for breathing it takes all my resolve and confidence to remain unaffected. So well done for how you handled it.