I think it can be done, judiciously of course. But I have no problem with my wife and her former boss having lunch. I have lunch with a female work associate, though we try to make it a group thing when possible. But if others in the group say their plans have changed, she and I will still go. We'll go out a night on occasion too - in group settings only. And yes, she's very attractive.
But here's a thought - why is it that I will only be interested in forming such a friendship with a woman to whom I am attracted physically? I still don't know the answer (do I have a hidden agenda?), but I wouldn't be hanging with this woman if she was unattractive to me.