Doing something as "familiar" as to borrow something of hers without direct permission might have made her wonder if maybe you were the type that would also think it was ok to go through her things as well and not think of it as snooping. I can understand your thinking on this but I'd have to agree with your friend and BF and say that the woman was justified in not asking you to watch her cat again if that's how she felt. In her house, her rules apply....that's why it's called HER HOUSE.
I'm pretty casual about stuff like this but I would have to say I might be taken aback by having this happen at my house while I was away.