I like the "past atoms" comment.
Most of the atoms on this planet are recycled over and over - only a small amount of "new" matter reaches earth from space. I don't "believe in" anything but I also don't rule out the possibility of some sort of "atomic memory" going on. Do molecules from a person become part of soil which nourishes grass which is eaten by a cow which is eaten by another person? Perhaps quite regularly. At this point, however, there is no reason to think (no evidence) atoms and molecules from a living being possess ability to "remember" beyond the death of that being.
So if it was real, that's probably how it would work (because there is even less evidence for a spirit world-afterlife) but pending evidence I see no reason to believe in it.
On a different but related note. We can't even trust our OWN memories of THIS life as accurate representations of true events. Remembering OTHER lives in the PAST would be a crap-shoot even if it was real.