Yes, I tend to agree that if you had some memory of normal reverting to it is not as hard.
But, I think generally speaking those that convert can feel more betrayed especially if they gave up worldly families and jobs and houses and romantic relationships and hobbies for the sake of the borg.
After burning that many bridges is can be difficult to reesestablish.
At the end of the day I don't think anyone leaving has the monopoly on suffering, it's not a competition after all.
Those that suffered abuse that they got no justice for and were revictimized by the congregation I feel for the most.