As a former "Cong. Secretary" ten years ago, I never tossed cards. I just took them out of the current box the C.O. would look through every visit. After about a year of inactivity, I took them out and added them to a stack of inactive/moved cards with a rubber band buried in the back of the Congregation Files. Reason? C.O.s would ask about every inactive publisher and push for a "resolution" during each visit. "Resolution" meant pestering the person with "shepherding calls" and unannounced "visits" with the C.O. afternoons. C.O.s had questions like: "How much effort have you Elders made to re-activate this publisher? Does the publisher have secret sins? Sometimes resulting in "Investigations".
If no one is bothering you, why wake a sleeping dog? Enjoy your life. :)
My 2 cents.