I would be worried if I was a father to let my daughter go into a public restroom full of women without me - but if you went in, you as a male might be a concern for at least some women in the ladies' restroom.
We don't sit down and have discussions. Trans activists seemingly would rather call people transphobes than sit down and have difficult discussions. I hope this trend changes.