Actually, Jesus didn't say that adultery was reason for divorce, he said that FORNICATION was grounds for divorce.
What's the difference?
You commit adultery by looking at pornography, but that isn't fornication. Adultery is simply the act of lusting for someone else, whereas fornication is when you actually act on that lust with a willing partner. If people were allowed to divorce simply for adultery, no marriage would stand. Of course, adultery is still wrong, but it is actual fornication that breaks the "one flesh" bond, and frees the person from the marriage. This is why Jesus said "fornication" instead of "adultery" as grounds for divorce.