There's something very strange about his obsession. In this video he seems to suggest that other people think he's only saying it because he doesn't have the figure to wear a skinny suit himself. In other words, he is acknowledging that this type of suit looks good on some men. He stated, "Some say you wouldn't look good in one anyway, that's why you don't want one", Tony responds "that's alright Buddy, I would never disrespect my heavenly father". Since he uses the term 'buddy' suggests this conversation is with men.
Either he's being hypocritical or there's only one other possibility that I can see. It's only other men that he thinks, find this attractive.