being "black balled" by any fraternity group is still a stigma.
Only to members of the fraternity.
Nobody else gives a $***, especially when the fraternity as viewed as lunatics or worse by the rest of the world. I spend a lot of time on a discussion board devoted to politics and military history. The guys there (for whom I have a tremendous amount of respect) all know I'm an XJW and all of them regard my exit as a a badge of honor and courage (they've used those words), which strikes me as over the top, but I'll take it.