I had the same experience growing up. But when my children were little, little, I did things a bit differently and had not issues with anyone saying anything.
I had a toddler and a new born. I would bring my toddler's kitchen booster seat with the tray attachment and let him color, snack, draw. When he got a little bit older, I had found cardboard bible story style books and allowed him to look through those. I didn't care that they were not JWs. No one said anything. I think I had the look of....I dare you to say anything.