We served in foreign language congregations where there were few children, or ours were the only one (eventually we had two) that were "born in".
Older and childless brothers and sisters, especially sisters, would sometimes feel the need to insert themselves in a "grandparent" role. My sons generally did not like this.
Your duty of care is primarily to your children, and you have to do what's best for them. Don't let anyone pressure you into something that you feel as a parent to be wrong for your child.