Until we figure out what time is, personally I don't think we can rule out time travel. Given our current understanding of it, I would tend to agree but really we have no clue.
Close to the atomic level there are several items that seem to defy conventional science and remain unexplained.
I don't rule it out, though lacking the ability to travel at faster than light speeds, time travel would be completely worthless. The problem with most people is that their minds are firmly rooted on Earth.
Consider you had a magical time machine that would transport you back in time. Where would you end up if you did? Essentially it would be reducing your personal speed to absolute zero. To walk you backwards through the concept.
- The earth is rotating, you would not be in the same spot on earth
- The Earth is rotating around the sun, you would not be in same path of the planet
- The Solar System is moving within our galaxy, you would run the risk of not being in the same solar system
- The galaxy is moving as well...
This pattern likely carries out as you continue to step back.
So for me, I would state that time travel is unlikely but science seems to have a way of solving previously "impossible" tasks the more we learn and understand them. For me, the first hurdle would be to reduce the effect of gravity on a body without impacting the effect of gravity on matter around it. Fun to think about, for sure.