The only option I see is me being stuck going along with him for a couple hours a week.
That's better for him and you than letting them at him and having to deprogram him weekly. You know what they can do to him in that time - they will take advantage of their position with him. Grandchildren are very open to the ideas of their grands. Don't ever let him out of your sight with them if you're playing the long game. Their relationship with him is important, but nothing like how important yours is, and if they interfere with that, they've crossed the line.
If you can cut it back to say fortnightly, you might be able to deal with it better. If you want to keep it weekly, what's the problem, are you working? Work shopping trips or outings into the schedule (eg "can't come over this week, we're too busy, but we're going to the mall, want to come?")
Edited: I kept reading:
I'm going to be asking their grandmother (who does love my kids dearly) to take them on fri night and have them back by sat night. that way there is no meeting to take them to. (unless its assembly day - then they wont go).she may take them in service, but i think i can handle that...they are very young still.
I'm just boggled. You're giving her a whole day, and they're going out on the preach. I've got to know, did you ever do that yourself? Were you never beaten up at school because you went out on the preach? Did nobody ever tell you that everybody who doesn't go to meetings is going to die at Armageddon and get eaten by birds? Boggled, truly. You're just giving your son away because you think your mother needs him.