The answer is YES it's not up to the Elders to decide who calls on some strangers door.
Put it this way your wife starts a study with him and his wife and the children sit in and her husband finds out he is a child molester....go figure.
They should NOT be able to go from door to door....EVER!!!
The Elders seem to think it's up to them to decide who is fit to call on your door.