In my opinion, I think it's likely that life arose on earth spontaneously. So it's reasonable to think similar life could do the same elsewhere. If they got a mere 100,000 year head start on us, they could easily have come up with better traveling technology than we have. If they are still limited to light-speed though, it seems unlikely they would ever reach us. There's always a chance, but it's so slim.
Even with faster-than-light travel, they'd still need to find us. What would you scan the heavens for, looking for a planet that contained intelligent life? The scent of Italian food? (Seems like a decent indicator of intelligence to me!)
For that matter, the past is a BIG place. Better civilizations than ours could have arisen, died off, and done it again many times and still not overlap with our civilization's time.
From what little I know, I'm currently thinking we haven't been buzzed by aliens. I suspect they're out there, but I doubt they know we're here.
Dave