I don't see how you can be letting your biological Grandmother down by going to her party. How is that letting her down? Because you have to leave at some point? This isn't the grandmother who sat up nights with you when you had bronchitis is it? This is one that has not been in your life for some time and wants to welcome you. But the bond is weak at best right now, and they'll be wise to not get grasping with you.
Did he say you are insenitive? Or did he say you are being insensitive? there's a difference. One is labeling who you are, the other is suggesting that you could handle it better. One is a better way of speech than the other. But either way, I don't think you are being insensitive to be there for her B Day and then to leave and go be with your friend.
Controlling people are very difficult and often wind up alone, wondering what the heck happened. If these are controlling people, then you are in line for some difficult times. Keep the reins in on your own life, conduct yourself like a lady at all times when with them, be classy in your manner with them, and do what you need to do, when you need to do it. Don't allow anyone to manipulate you.
Try to enjoy them on whatever terms work for you, and remember that you have had a life apart from them too. You'll be okay.
H.S.