One year I was really sick with a highly contagious bug and was under doctor's orders to stay under quarantine. I got to miss the Memorial that way. It was awesome.
You could always say that someone treated you like crap at the Memorial and that you're never going to set foot in the KH again. Then say: "Ha! Gotcha!! Had you going there, eh??" They'll never ask again after that.