It's an illusion. You take a gray square and put it in a white background. Take an identical gray square and put it in a black background. The one in the black background is going to look brighter, almost white. The one in the white background will look darker. You can also see this if you have black squares with white space between them, running up and down in a grid. Where the white spaces intersect, you will get the illusion of a gray spot there.
The same thing happens when you put some moderately attractive people in a group where most of the people are ugly (those old fogeys with zero tolerance for fun). Bearing in mind that the stress of being a witless wears people down, and hence destroys this attractiveness faster. This moderately attractive group in the Kingdumb Hell is bound to look more attractive, like your gray square in the black background. Often, the pickings are slim--meaning it doesn't take much to impress others in the congregation. Bearing also in mind that they are programmed to look down on worldly people.
Put those same people in the world, and they don't look so attractive. In fact, they could actually be below average. This is like putting your gray square in the white background. The bigger the group you are comparing these "moderately attractive" people against, the better the chance they are going to look very plain and unappealing. Besides, in the world, you don't have rules against going outside the group.