I can only think of two examples why someone would be afraid of other human beings:
1) They can cause you or loved ones serious bodily harm or death because you are defenseless against them for whatever reason
2) They have the power to influence others to cause you or loved ones serious bodily harm or death against which you find yourself defenseless.
Since a CO would doubtfully bring about either why be afraid? The most he can do is threaten your psychological well being or create a harsh social environment. Both of these are circumstantial/environmental and possible to overcome. It's up to you to change your circumstances even when the prospect of removing yourself from these circumstances/environment seems daunting.