what if a dog had come out to meet him? To offer up an unclean animal would be an affront to God, so Japtha most likely dedicated her to service to the Lord
It seems there's a couple of flaws in that reasoning.
I haven't read the LDS source (I'm on a bad connection right now) but what justification does it have for suggesting that the Jews at that time would be keeping unclean animals in their houses? Even many muslims today do not keep dogs in their home for similar reasons. Would Jephtha really be expecting an unclean animal to come running out of his house to greet him? I know there is reason to believe he was the son of a prostitute, but still? Even in the account of Jezebel, she had to be thrown outside before the dogs could get to her - they were not invited into the house to kill her (yes I know it was supposedly a "prophecy" so it had to happen that way but...).
Assuming that Jephtha could have expected and conceivably there had been a dog running out of his house and such an animal would not have been an appropriate burnt sacrifice, are you then saying the same dog, still an unclean animal, would instead be an appropriate servant around the tabernacle?
Judges 11:31 in the Jephtha account (apologies that it's the NWT) says: "whoever comes out of the door of my house" - I can't look up a Strong's Dictionary right now, but it seems apparent that he was limiting it to a person ("whoever") and it was not something running down the garden path, but definitely 'out of the door of his house'.