Apostasy is not grounds for divorce, but many times people will lie and say their spouse confessed to adultery when they did not. Especially if it is a man they can and do get away with it. I know of a case where a man claimed his invalid wife confessed to an affair many years previously. He got divorced and married his wife's much younger caretaker. There was a lot of gossip, but he was a powerful elder so he got away with it.
I say get on with your life and let him do what he wants. Don't lie or say you did anything you didn't, but if he wants to claim you cheated, don't go out of your way to contradict him, it's not worth the bother.