“Pagan” was simply the Israelite term for the rural poor who were left behind in Canaan and Jerusalem during the Babylonian captivity.
since those who were taken were the upper classes, they looked down upon the poor rural Jews they returned to, because their religious ideas had now diverged. The Babylonian captives had absorbed religious ideas of their captors - for example the concept of light and dark, of absolute good and absolute evil - and had lost the anthropomorphised god with human frailties (such as jealousy) that those left behind still retained.