I can sympathize with you Looloo :( We seem to have very similar stories.
For me... I can only imagine the talk that is going on behind our back! I'm at the point, I just don't care anymore. If they use me as an excuse to why their relationship with the kids is almost nonexistent, then that's fine as long as it keeps them away from making them feel less then or bad for how we choose to live our life. We keep our visits to a min these days and it's working for us.
I wish you well.