I would go...and take your brother. When the JWs start acting like assholes just tell your dad that you are sorry but you have to go, and that you thought this was a FAMILY reunion and that FAMILY members were getting together and not a "congregation picnic" where only JWs were worthy of participating or being treated civilly. And that if you wanted to be abused by holier than thous, youd go back to meetings.
Its nice that he asked you to come...but I see a whole bunch of strings attached to that invitation.