It's almost like they wont even let me mourn the loss of someone who was very special/close to me.
They don't see that you have a right to mourn with them. Many also think that when we leave the organization, we leave all loving feelings behind or that any we have are clouded with guilt.
Really tough to know how they would treat you personally if you decided to attend the service at the hall. Seems to me that you should go - because you *want* to go. Maybe give a simple, non-preachy card to his wife and maybe also to other family that were close to you.
If you decide not to go, and/or if it doesn't give you the comfort and closure that you desire, you can always do a little ceremony by yourself or with your family. Something at the beach, in the mountains, or at his gravesite.
Really sorry to learn of your loss. Sounds like he was a good man.
-Aude.