If he mentions anything to you, like the nauseating "We've missed you..." followed by a long pause waiting for you to explain your absence, just respond in turn "Yeah, I've missed you all too." (May not be sincere, but neither is their nosy "We've missed you" polite interrogation tactic.)
If he says "We've missed you.."
Give him a smile and say "Really, how sweet. You should have come round sometime."