Jeff,
As you know, I don't have any legal credentials, but I do have some experience with similar situations.
As far as I know, without paternity, he is just as you described; a virtual stranger. I would highly recommend going the "establish paternity and visitation if you want to see the kids route." Don't be a hardass about it or anything. Just tell the guy you have two choices, to allow him in your home or to allow him to take the child... he will likely present alternatives and those might be reasonable.
But the bottom line is if he wants to wear the daddy-pants, then he needs to step up and sign up. The nice part about this is establishes guidelines, and it also FORCES HIM to actually do something proactive in terms of taking care of his children. If he is serious, then a couple grand to a lawyer will show that. If he isn't willling to do those things, then it will show he is qualified to own a dog maybe but not for parenthood.
Try to keep it amicable though. Biology takes precedence from a legal standpoint. He could mount a legal battle to gain custody. I am not sure that you could do anything about it, and it would be unfortunate for the other children. I don't know about bio grandparents, but in the case of non-bio adoption you pretty much have to prove the other parent is unfit! Not an easy or inexpensive thing to do legally.
I would do everything you could to keep the other children out of the loop whatever you do. They don't need to know about any visit from other baby-daddy's.
Just give them all as much normalcy, discipline, and love as you can. Whatever you do, do it with their best interests in mind. As long as you do that everything should be ok.
Good luck. Sorry about all the drama. Take care.
CYP