should i be worried?
I would be, but looking at the bigger picture. Was his relationship with you the reason for him getting disfellowshipped?
Also, you should be able to just ask. I think the suggestion of asking if you could go too is good, but (IMO) that should be a second question, not the first. Trying to find out how he feels or what is the rationale behind him going to the memorial here is only guessing.
Talk to him about it. He may need to know that you are supportive of him.