It seems that there must have been much ado about the origins of his birth even during that time.
This is from the Gospel of Nicodemus at the time Jesus was brought before Pilate:
3 The elders of the Jews answered and said unto Jesus: What shall we see? Firstly, that thou wast born of fornication; secondly, that thy birth in Bethlehem was the cause of the slaying of children; thirdly, that thy father Joseph and thy mother Mary fled into Egypt because they had no confidence before the people.
4 Then said certain of them that stood by, devout men of the Jews: We say not that he came of fornication; but we know that Joseph was betrothed unto Mary, and he was not born of fornication. Pilate saith unto those Jews which said that he came of fornication: This your saying is not true for there were espousals, as these also say which are of your nation.