I have a similar situation in my family. My younger brother is 29 years old, has a closed head injury, and is mildly bipolar. He has a hard time controlling his temper, and has gotten violent with my parents and with me. My Dad often excuses this behavior, because he *can't* control it. I know that he has problems with impulse control, but I also think he needs to see that there are consequences for his actions. My parents are in their early 60's, so they are getting too old to have to worry about my idiot brother attacking them.
My husband feels exactly as you do- if my brother ever attacks me in front of him, he will kick his ass. I guess it depends on the person- in my brother's case, he is a total manipulator. He gets away with everything, including not having to work, because he *can't* help it. But he can mooch free meals off my parents and spend all of his money on pot. OK, I'm venting now. But I know exactly how you feel. I don't think you were out of line.
kj