I did a short internship with a county Department of Assigned Counsel-Dependency Unit..not really familiar with the laws...only slightly. You have a serious situation on your hand. Most states have laws mandate that their first priority is to keep the family unit together. In this case, the "family" is your daughter, this gentleman, and your grandchild. You MUST seek to obtain legal guardianship of your grandchildren.
As the biological father, many states will be obligated to grant him visitation and/or custody should he legally request it. However, you can raise questions as to his fitness to parent, i.e., is he a drug abuser? If so, he will have to provide UAs to prove he is no longer using. Does he have a history of violence? Then ask for supervised visitation. This most likely WILL trigger involement by your local Department of Social and Healther Services.
Your daughter's opinion will play a large part in any visitation/custody arrangment also. Hopefully, he is unable to manipulate her negatively. I can not over emphasize to you the seriousness of this situation. Please seek legal counsel.
May Jehovah bless and protect your family.