They justify it like this:
When the scripture says "it will be when no one is expecting it" that refers to worldly people.
Another scripture is used to say "keep in expectation of it, it will not be late"
So, in other words, "True Christians" will be in expectation and ready for the day, while "worldly" people will not know the time nor be ready for it...
Make sense now?