Practice a few stock phrases that you can use as appropriate.
Oh wow, I wish I could, but thanks for the invite anyway....(don't elaborate). Smile!
It's just not a good time for me right now, but thanks for the invite. Produce a (very slightly) apologetic smile.
I wish I could, but I have another commitment (must see TV) but I will get back with about a better time, apologetic smile.
The main thing to remember is you owe these people nothing,. They don't give a rat's a$$ about you, they are just being guilted into going after the missing, so don't feel guilty yourself and you will break the vicious cycle in play here. The minute they break you down into an explanation they own you, because then they can judge if your explanation is worthy. Don't give them that power.
I am an introvert, it took me thirty years to learn to do this, until then people freely took advantage of me, so learn from my mistake, it will be very useful later in life.