Grandparents most certainly have rights. In NY State grandparents have standing with the court if there has been on going relationship with the grandchildren the and if they pursue their rights then they might well get visitation. I don't think it would be hard to make sure it is supervised visitation, with the parents or other neutral party supervising.
My son has been in a four year family court battle and my husband and I had to petition the court to get visitation with our grandsons. So far , we have been granted 2 hours every other week supervised . It is an expensive nightmare. Soon we will be getting unsupervised day visits but our ex daughter in law has fought every inch. She has no problem lying under oath to get her way. Religion has nothing to do with our case though.
Much trouble can be saved by offering some small access, 4 or 5 short day visits a year, possibly in your home, would satisfy the court. I don't think they would give much credence to the shunning but would see it as not being reasonable or cooperative.
I definitely think you should always be present when your parents are with the children, it will stop many problems before they start and you would have a record for the court that they only had supervised visits.
You would have to check with the judicial system where you live for details.