It's not the gifts and attention that make holidays/birthdays special for long term childhood memories but I think it's the socializing with other kids both at the event and the before and after discussions. Not celebrating those milestones or holidays tends to separate witness kids from the majority of their classmates and that lack of social integration can have a negative affect later one.
I am sure not everyone is affected the same way or at all. All I knew with my kids was that life is tough enough without me adding additional stress into the mix.