I think it's a generation thing too.
As a kid I used to sit with a friend on a cushion in the very back of Dad's estate car (station-wagon) with no seatbelts while the grown ups sat in the 5 seats. No seatbelts in the rear but two in the front. We would travel on holiday or to assemblies like this - it was the norm.
Then when seatbelts and insurance limitations got stricter we might risk it for short journeys. Now I won't drive without a seatbelt done up and I won't carry anyone in the very back (if I had an estate car). It's legality and liability as well as safety that bother me.
I personally don't see a huge risk (in chances of a bad enough accident terms happening on that rare unbuckled journey) in riding unrestrained but it's a good habit to get into.
But it can get ridiculous - I had a car where we went to pick someone up and the seatbelt clip broke as one of my kids tried to buckle up. So I have a choice - make them walk with me or my wife for five miles home or take the chance. I took the chance and would again. But got the belt fixed immediately and only used the 4 working ones until it was repaired. My current car is 5 years old and has airbags too - the side impact ones are not a legal requirement and if one packed in I wouldn't worry about replacing it.
In the OP's situation with other peoples kids I would stand my ground or offer two journey's if it was a short hop like you did. No one - especially the car owner - should be forced to go against their personal wishes on health and safety.