I'm sorry to hear about your troubles.
Don't let anyone talk down to you for marrying a non-believer. We all make mistakes and are prone to continue making them. Who stand tall before God? No one!
The very best advice I can give to you would be to first and foremost DO NOT GO NEAR ANY KINGDOM HALL. Sorry for the capital letters but I cannot stress this enough. You and I know what they will recommend you do (field service, pioneering, attend meetings, studying, etc.).
Plan a date or a time to speak to him at home over a cup of tea, coffee, whatever, and pour your heart out to him about your concerns. Tell him you love him and want to make it work. If he is a non-believer do not push religion into the matter. Although you could read some Psalms and other texts as marital guides but nothing else and let him know so. Kinda like consulting a manual or reference book.
Pray to Jehovah God and ask him to give you peace and tranquility and to help your marriage survive. If speaking with him does not help, try to find some counseling or a mediator. But don't worry yourself to death. Instead look to the positive side of your marriage and try to build on these.
I'll be praying for you, your husband, and your home.