He mentions to her that his master entrusted him with "all his belongings" - that is my view as well why he did not accept the "invitation". A wife has an owner. Wen married, a women is 'owned' by the husband. Like he would not date to steal anything else that belonged to the master, he would also not take the woman.
I do not believe it was because of the sexual morality as in those times, where were no specific rules, But ownership was already a familiar term.