The first time I missed the memorial, I was sick and couldn't go, and felt guilty because that meeting was so important.
Then another year, I missed again, this time I wasn't sick. I still felt guilty, because I still believed in the religion to a certain extent. I didn't go because I didn't want to go back to the meetings, and feel trapped once more.
Maybe some feel guilty that they aren't doing anything for the Memorial or Easter, depending which religion you think is right, because after all Jesus did say to remember him.