Do what others have suggested. Dress business casual, show up just for his talk and leave right after, etc. But I'd also suggest making it clear to them beforehand that this is what you're doing and that they are not to construe your attendance as a hopeful sign (they will anyway). That way, after you go and they start pressuring you to come back even more (which they will) or that you will go to any of his future talks, you can just say, "No. I told you before I went that I wasn't coming back. I told you I was only going as a kind guesture. You've been kind to me this past year and I didn't want to hurt your feelings or be rude. I still don't, that's why I don't want to discuss this any further." Then don't.
My parents were never JWs but they belong to one of those fundamentalist megachurches (their pastor is on TV) and are always trying to get me and my brother to go with them. We won't but then neither of them have parts on the program so there's no "come see *me*" thing going on.