I find these sorts of experiments interesting. The idea of "group selection" has been around for some time, although most biologists now use the term "multilevel selection", or they're more specific like "deme selection" or "clade selection".
I'm not sure what study Carl Sagan was writing about. I have yet to read his book Broca's Brain, does that sound like its it? Anyways, the study sounds roughly parallel to what's called the iterated prisoner's dilemma. Basically, if its likely that a prison would never deal with someone else ever again, they'll sell them out (cheat) if given a tempting opportunity (being set scott free). But if its likely that the person will be called on it in the future, either by the same person who was betrayed or his companions, then its better to cooperate with the other prisoner and both of them say nuttin' to the feds. Ratting on the other might get ya whacked. So in a way, cooperation can logically develop even on a basis of self-interest. If anyone likes to read more about it I think Wikipedia had a nice writeup on this.