I have sympathy with your position with a few provisos.
I feel no need to back off in challenging factual claims made by religion. If somebody wants to argue that religion might be/has been useful even though it is not objectively true I can live with that. When fundies want to argue that the bible is inspired or that god heals the sick or that creationism is scientifically credible then I am your opponent.
I still have no tolerance of Islam. It is the slow kid in the class. If it ever catches up in terms of personal freedoms then and only then will I give it a break.
Having said all that, I still have concerns that religion was a force for intra-group and national cohesion and its demise is not an unmitigated good.
Many liberal xtians and reformed Jews have found a way to hold on to religion as a social influence while avoiding inter-group animosity and without lying to themselves about what is true. Maybe there is something to be said for that.