Simon, if you can wrap your mind around your reasoning in the OP, then why not just replace universe with God, and use the same reasoning as you did for no time before the universe?
That works. Either way god as a concept is destroyed and any god you can destroy isn't much of a god.
As Hulk would say "puny god".