M y condolences to you and yours in this troublesome time.
I have no certain information but I firmly believe you should contact an attorney who specializes in these matters.
Perhaps you should either adopt or be granted legal custody of the children?
The past and potentially future instability of the father(s) does not bode well for counting on them to be "reasonable."
Protect the children and yourselves.
Best wishes.