We have also had invitations. If you do not explain to them why you won't attend, they will just continue to invite you. If you feel comfortable, speak from your heart. Be honest.
We have had to do this with our family. We told them we love them and nothing will change that. We will be there for them BUT if you don't believe in the doctrines, tell them.
You have one advantage, you are not baptized.
If you dodge them, they will just come back to you with more invites.
If people WANT to be in your life, they will, If not, at least you were honest with them. Good luck. Let us know how it goes.