We did allow them the time with the kids until the kids started coming home saying "Jehovah doesn't like that", "Jehovah doesn't like birthdays...Christmas".
Same here. My 3yo daughter is secretly palmed off to my mother (who tells people im dead) by my ex-wife. At xmas time she randomly told me "Jehovah doesnt like Christmas". Fortunately, i acted surprised and said "doesnt he? well thats alright because i like it. Do you like it?" and she said yes. Now i say "it's ok, Jehovah's not real. He's just pretend".
I think your boy would be better off without them. Think about it, they have a toxic belief system. If your boy ever did join the JWs (heaven forbid!) and then left they'd all drop him like a hot brick.
My family haven't spoke to me in 2 years but nag my ex wife to see my daughter. She doesn't really like going there but my brother tells me she's been handing her over every couple of weeks. As long as the kid is with you more than the JWs i think you'll be ok provided you call bullshit on any JW crap they come out with. And reason with the child (something JWs wont do, allow critique and inquiry into their doctrine).