It didn’t always mean even a loss of privileges. In the late 70’s when the beard thing broke we wanted to appoint to be a MS a bro who was bearded. The C/O at the time didn’t like it , he said it was just because he was smart & educated and we wanted to park the account’s position on him ( truth is he was right about that)
Anyway it went up to Bethel for ratification and they agreed it , with knowledge of his beard. He held the accounts servant position for years.