If I remember rightly, the four ages of man are:
Lager, Aga, Saga* and Ga-Ga.
So, on that logic, you stop being called "young man" when you become more interested in cast iron stoves, and (slightly) less interested in booze.
Hope this helps.
iggy.
(*Saga - for the uninitiated, a company specialising in selling holidays and insurance to the over 50s.)