Actually, yes, it would matter to me...
You see, I'm trying to keep hold of the best possible relationship between me and my devout JW family. A very long time ago, as a child, I figured out that I wouldn't live happily as a Witness. That being the case, I never got baptised, but maybe I was being disingenuous...I always gave, and still give, the distinct impression that I am somewhat interested in the JW faith, but am holding back for some reason.
This act, this...falsity...is what keeps me within a semi-normal relationship with the people I love very much. So, I've been able to keep any JW 'label' from applying to me, and therefore giving any reason for my JW family to shun me. I aimed for the 'cracks' in this sick JW belief system, and somehow, I seemed to have hit the bullseye. I have seemingly locked onto the only 'semi-honorable' exit from this miserable cult.
So, YES, I would care if any label were to be slapped on me. It would mean strain on the relationship between me and the JW family. I'm not sure exactly what power the local Elders wield, given that I was never baptised, but that seems to be a crucial point, for some reason...