Your husband sounds quite understanding and forgiving. His love for you is saying "Honey, I love you enough to lose you if that is what you want...". Quite a man. It sounds like he also realizes the push-pull of rebellion. Anytime you tell your teenagers they can't have something, what do they do ? So, instead, talk to them gently, let them decide, and accept them for the choices they make. It sounds to me like your husband will always love you... regardless of the path you choose.
Now, this "in love" thing... How many defenses have you dropped around your affair partner that you maintain around your husband? That's are really hard question to answer honestly, and may take a counselor to help you sort it out.
Also, how old do you feel around your affair partner? Did you seem to age regress when this feeling started? To what age? Teenage perhaps... flirting, giggling, breaking taboos? If this is true, your husband can sense it, and your defenses around him are probably getting a little harder to penetrate.
And, finally, I would ask you... what wounds did he let you expose, without criticism or complaint? And is the "romance" really reciprocal? Is he disclosing just as much as you are, without being asked?
These are tough questions... there are many more to ask yourself. And there are people that accept poly relationships. If it is not against any of the participants morals, then with open, honest communication may you can all live together?
One last thought.... You may be his "out" from a bad marriage also. Personally, I would stay clear until he quits justifying his actions and talks with his feet. Just my opinion.
Good Luck... you are not in an easy position to get out of regardless of your decision at this point.