This man has lied to everyone. The truth is he is a member of a high control group and part of the culture he belongs to does not permit him to be friends with anyone outside the group. He is only allowed to date someone if he intends to marry them and sex is not allowed outside of marriage. If your relationship was known to other members of the congregation he would be censured and may even be shunned and thrown out of the faith.
He wants everything, to have sex with you and to maintain his position. He can't have both and be honest. He can't see you as a serious relationship because you don't meet his religions criteria and he does not care enough about you to put you first. He wants you to make all the compromises whilst treating you as a shameful secret. If you continue you will be constantly expected to be available for him whilst at no time will he make compromises for you. Christmas, birthdays and thanksgiving are not important to him so you will spend these on your own.