John doe you are correct, but you are basically describing the same scenario that I described whereas more powerful men get wives and the less powerful do not.
In a scenario that you described, you suggested that only rich men could afford wives so unless thet are allowed multiples, then population growth would slow down. However, if those wealthy men were allowed only one wife, do you really think there would be all these leftover women that remaned childless there whole lives? They may not get married but they would still be having children, with the poor men, that's just human nature. The idea that poor men and unmarried women would remain virgins there intire lives just isn't believable.
Consider also cases where a very wealthy man has vast numbers of wives, I think the chances of any women getting pregnant decreases with each new wife. Think king david here, do you think those 1000 wives and concubines each had good chances of getting pregnant? Each of those women would have been very lucky to have just one child, but if they had there own husband they could be having a dozen.
Polygamy does not increase the rate of population growth, it simply increases the population of the polygamist.