Archaeology and paleoanthropology show that violence was much more prevalent in prehistorical bands and groups, than it is today. Violence between rival bands is also very often in some our closest relatives today, the chimpanzees.
When the first archaic members of Homo sapiens set foot into a new territory, one of the first things they did was exploit to the point of extinction many other species of animals and plants, including other hominids, and even other human rival tribes. That was the rule back then. (and yes, man-caused extinction is a really old habit.)
The "noble savage" notion of Rousseau has been proven today to be a total myth.
Organized society still holds to many injustices and has not stopped wars and violence worldwide; still, in general it is a lot safer world (and time) to live, than living in pre-organized times and places. (This doesn't guarantee, of course, that things will get even better in the future; study of the past indicates, it doesn't prophesy.)
The evidence indicates that while peace-keeping and altruism are part of our nature when it comes to a tightly tied group like a family, when it comes to relations between groups, violence is our natural tendency. The point of having superior intelligence to other animals is to be able to overcome those bad traits of our nature.