I would say that the fruitage would be the result, not the effort. Someone could spend a lot of time in a soup kitchen making soup and filling bowls, but if no one comes in to eat then they're working but not yielding any fruitage (or the fruitage they do yield is wasted soup - not something to brag about).
The point is that the fruitage is somewhat out of your control. Going back to the "I planted, apolos watered, but god kept making it grow" concept - the fruitage is evidence of divine blessing because it's not entirely in the hands of the ones doing the work. That's my interpretation of it, at least. So applying this to JWs - they spend a crap ton of time in the ministry, that's their work. Their fruitage is that they grow at roughly the rate of population growth, which does not indicate to me any sort of divine blessing. By their fruit you know them - they're a waste of time.