I wouldn't get too caught up in it. We're doing much the same. Nearly the identical situation.
Here's how we look at it. The sprinkling in attendance for us is not for anyone else or because I'm compelled to do so. It diffuses the situation.
Lastly, have fun with it while you're there!. Believe it or not there are things that you can say that will rattle those that pressure you where you can almost let them know I'm not buying this mess anymore but at the same time they can't pin it on you.
I'll give you an example I had happen not too long ago. A brother who is the type that is always putting on the guilt trip was gloating at how the organization is so exacting regarding the date of Jesus memorial.
I replied casually with..."you know I've always wondered how that works out given that us celebrating the memorial on the 23rd of March, would make it impossible for the Festival of Tabernacles, a holiday Jesus attended, be celebrated on the date commanded in Leviticus 23".
Now I've done a ton of research on this topic and so I'm rather certain of it all. There was no place he could go that I wasn't prepared for. The look on this brother's face when I said this...priceless. He wanted to lash out but I asked it in such a humble I don't understand way but with naming scriptures and pointing out this festival was clearly important to Jesus what could he say?
He replied with..."well I'm sure they have people a lot smarter than you and I at Bethel. I said well they certainly have the time for study since we forego college...what a blessing". See I took what he said and made a subtle dig that none of them are college educated.
I've gotten to the point now where it is just so easy to say things in a way that they can't pin anything on you but you can make them think...and think hard.
When you're armed with truth there's not much they can do in the pressure department. Just try to form it in the form of a question. Make a game out of it.
By the way this is not my tactic. I can't take credit for the "make a game out of it" mentality. Another ex'er told me this when I first awoke. It has been fantastic for me as I too get pretty bummed out with it all myself.
So have fun with it especially with those that take the hardline, pressuring you with guilt! It will actually give you a chuckle.