I don't feel you were unbalanced or inappropriate in your comments but then I am no longer part of the organisation. It seems that this is an excuse to grill you about non attendance and to fish to see if you have leanings towards apostasy. A fade is ok but if you associate with witnesses socially it seems to trigger action as you are seen as a potential problem.
I would say that you appreciate their concern and you will certainly look into this in more depth as a family but you would decline the offer of a visit at the moment as you are very busy and cannot commit to a specific time. If you feel that you could benefit from their input then you will of course contact them. Inviting them into your home can be a recipe for disaster I speak from experience.