Yes they have the power to control you in this matter they can also decide whether you will conrinue being a publisher or not and pretty much they decide what goes on with you of course someone has to open his mouth about things you said or did.
If they think you are not putting much into field service they can decice to terminate your service to God which is ridiculous well that's how it is.
In my experience a friend I met at the hall was an unbaptized publisher for a while, he was also commenting at meetings he was on his way to water baptism but all of a sudden he was announced to be no longer a publisher. everybody look down on him from that point on nobody talked to him no more I was the only one I even invited him to go to field service together and he said yes but he didn't like going with my group. I was new at the hall so everybody treated nice there was me a young brother baptized publisher who was born in the truth, they even try to set me up for marriage with an elder's daughter, I said no thanks, her dad grew angry at me from that point all her family together grew angry as well. For no reason they started treating me bad but I pretented I wasn't totally listening to them and instead had a smily face with them, The elder try to take me out of the hall and then I befriended the president the oldest elder at the hall and they stopped for a while but nothing is the same again they don't see me as a brother just because i didn't marry the sister.
They can use anything against you is up to you guys to battle forward not letting any of that stop you from being loyal to God humans will always try things on you just don't listen to whatever they say about you.