Sure, you're allowed to get together with the brothers and sisters. However, if anyone has a good time, he or she is setting himself/herself up to be "marked," "reproved," "disfellowshipped," or some other form of being regarded with suspicion.
Most of us adults accept responsibility for our own behavior because we don't have masters to explain to us what is proper and improper. If you're all grown up, you should know how to behave yourself and not accept being talked down to because you had an evening of relaxation and/or amusement. Life's too short for this nonsense.
As Sophia Loren used to say, "Wear my perfume and let them talk." If you have someone talking disapprovingly about your behavior, I envy you.