The "imperfect men" argument by itself is not really that bad. The "secret society" I joined to slap the Washtowel Slaveholdery in the face is also run by imperfect men.
The difference is what happens if the organization makes a mistake and the members see it. The "secret society" actually encourages members to find the flaws in it and improve it--I found a couple, but they were generally very minor problems in an otherwise sound organization. They do create more value than costs--which is what I define a good organization is. By contrast, the witless organization doesn't allow any criticism, not even in minor issues. That allows a Filthful and Disgraceful Slavebugger class to develop at the top of that organization, creating self-serving doctrines. The result is that the cost of being in that organization exceeds any benefit. And that is why I bash them, not because they make mistakes.