It derives from the ancient legal concept of expiation. This was a provision that allowed one's legal guilt to shifted to another person or creature. Punishment for the crime is still meted out, but it is the animal or person that substitutes for the criminal that receives the punishment. In the Torah, a scapegoat (hence the idiom) serves this purpose. It doesn't make any sense in modern concepts of human and animal rights, but this was a popular provision in the law that allowed people to escape fairly strict punishment (e.g. lex talionis, as found in the Code of Hammurabi, the Torah, and elsewhere). Since human law was believed to be an extension of divine law, this provision was applied to religious offences (hence, the Jewish concept of atonement). Expiation is a later development in the practice of sacrifice, supplementing the older practice of propitiation or appeasement of the gods. I think it is important for one to realize that the Christian concept of Jesus' sacrifice is an expiatory one, not a propitiatory one. It is not a matter of God needing to be appeased, but a matter of an abstract legal debt being absolved.
This builds on a much older expiatory view of suffering and martyrdom found in Deutero-Isaiah and the Maccabean literature (cf. also Daniel 9-12), which themselves built on the Deuteronomistic view found in 1-2 Kings and Jeremiah that explained the destruction of Jerusalem in 587 BC as resulting from the collective sin of Judah against Yahweh. The prophets saw a future restoration of Israel into God's favor, but how would that occur if Israel is guilty? Deutero-Isaiah explains that the faithful remnant who suffered in exile were offered up in expiation for the collective guilt of the whole nation. This explained the larger theological question -- Why should the righteous suffer for the deeds of the unfaithful? Why should Ezekiel and others go into exile when they lived righteously? The Deuteronomistic perspective claimed that these calamities were punishment for sin, yet the righteous did not do anything to deserve the punishment. Expiation supplied an explanation: They suffer on behalf of the many who sinned, in order to absolve the legal debt of the whole nation. But after the exile, Israel was not restored the way the prophets foresaw. The Hebrew apocalypse in Daniel (ch. 8-12) showed that the Persian and Hellenistic periods remained "difficult times" but promised that the foretold restoration lay just ahead, following the Maccabean crisis. This crisis saw many traditional Jews killed for holding fast to their beliefs and traditions. The Hebrew author quite consciously reinterpreted Deutero-Isaiah to refer to his own time, and explained that these poor souls were not dying for nothing, but that their deaths meant something -- they were accepting the legal burden of the nation and atoning for its collective sin. The later Maccabean literature (especially 4 Maccabees) developed this idea along lines that directly anticipates the Christian conception of Jesus' atoning sacrifice. Meanwhile, the Jesus movement universalized the old hopes of a revived, restored Israel into a gospel proclamation that the whole world will, or has been, set right to God through Christ.
Paul however had another ulterior motive in pursuing an expiatory understanding of Jesus' death. He recognized that, as a legal provision, expiation was the undoing of the whole Law itself. For, if it was really possible for anyone who so chooses to be set right before God for all time (i.e. justification), there would be no need for the Law to provide expiation to do the same thing temporarily (i.e. through the sacrifices described in the Torah), it would become superfluous. The Law itself provides the very thing that would render it null and void. But this could not happen until Jesus came, for only he could truly substitute for the whole of humanity. Why only Jesus? Paul elaborated on theological grounds the uniqueness of Jesus in relation to God and to man.
Note that all of this only makes sense within a very arcane, ancient legal provision that no longer exists today in modern jurisprudence, at least in the same form. Outside of this context, it makes no sense at all.
