My wife and me did, for the same reasons as you, and never gave it a thought, apart from being aware that other J.W's would have a stupid fear of doing so etc. I never bothered about what other J.W's would think of our enjoyable visits.
After we woke up and left, we were able to visit more of these wonderful buildings, and we also visited Winchester Cathedral one Sunday, and joined in the Service to the extent of sitting through it, we really wanted to fully see the cathedral when the Service had ended.
We also visited a number of Churches and Chapels, of various Denominations, and sat through the Services there.
Nowadays we would not bother with the Services, we are both Atheists. But if able to do so, will still visit any building that is historically and/or Architecturally interesting.