They say it's because it reflects well on Jehovah's name, but it's really about control, control, control. I really think the old guys get off on being able to control women especially.
And to the point on the impression it makes on 'worldly' people, once I stopped at an elderly relative's house after service with my young daughter. We were both wearing skirts. My relative, who was a church-going, conservative type person, remarked "You're dressed up all the time. You look ridiculous".