You should not have to take steps to avoid him. Treat him like any other customer, and if he tries to talk about personal matters just say, "Is there something business related I can help you with?" If he persists, just say, "I appreciate your concern, but if I wish to discuss personal matters I'll do so with my husband, not with you."
If that doesn't work call the police. You do not have to tolerate harassment in the workplace.
W