Es, it sounds like you are better off without them. Although having grandparents is nice, it would be worse if he associated with toxic grandparents. You sound like a good mom. Sometimes grandparents have a tough time seeing and admitting where they are wrong so they have make it out to be like it's your fault. But its not your fault it's theirs.
My mother thinks that it is her childrens fault that 75% of her kids and 90% of her grandkids don't talk to her. Of course it couldn't be because she is a lying manipulating backstabbing drunk... no no that's not it.
Hugs ((Es))