There are reasons people have panic attacks. Panic is physical reaction to either something that you fear or it's a reaction to a dangerous situation. Your subconcious KNOWS that you should not be there even if you don't realize this. And your subconscious is making you react in this manner so that you remove yourself from the situation. These feelings are there for a reason and you should take them to heart.
Much like standing at the edge of a cliff or the edge of a tall building makes me have a panic attack. That's my brain telling my body to get the hell away from the edge because I could fall and die! And the only way for that feeling to go away is to remove myself from the dangerous situation. This is natural. Follow your feelings. That's why God put them there.
If your husband loves you he will understand this and he will need to respect your wishes. Put your foot down for your RIGHTS and BELIEFS! Don't feel that you HAVE to go. You don't HAVE to do anything that you don't want to. Try turning the tides on him. If you have any hobbies that are usually something only females participate in, tell him that HE HAS TO DO THIS WITH YOU! And then see what he says. If he doesn't like the fact that you're forcing him to do something doesn't like then say, "See? Sucks doesn't it? Now respect my likes and dislike and don't force ME to do something I don't like."