I remember a time in my life where the congregation I was attending was discouraging me more than anything else. I was coming back from there less happy then I was before going.
In the end, I believe it was simply a matter of compatibility as some of my friends told me very good thing about that congregation. Anyways, I moved to another congregation without changing territory and I remember the elders were not happy about it, but I did tell them the truth.
Anyways, I was than very happy in the other hall for quite a while before I had my kids and my crisis of conscience.
You should not leave the JW because you don't get along with people in your hall. You should stop however if, in the end, it violates your personal beliefs and conscience. So until you reach that state, if you ever reach it... you should definately change congregation.