This sort of problem has been an issue for me too.
In my senior year I had a car, so I was able to drive to school. A (former) friend's younger sister also went to the same school and asked if I could pick her up for school. My family and my friend's family went way back, so I didn't consider it any trouble to do this. After a week or so of taking her to school, her stepdad approached me and asked me to stop giving his stepdaughter rides because of the controversy it might create. I didn't think it would cause any...but I agreed to do this out of respect for his wishes.
Edit: At least with my story, the guy didn't threaten to go to the elders. We settled the matter as men and let it go no further. Thank goodness for Liberal Dubland milieu...