Remember there is a difference between being clean and being sterile.
There is far too much emphasis on sterilisation in homes (kitchens and bathrooms), etc, with the use of chemicals that I can assure you are much worse for you than any bacteria growing on your "sweaty balls."
I often will eat something if it has fallen on the ground (soil or dirt) and just lightly dust it off, there's nothing wrong with keeping your immune system in ... action.
I'm guessing this recent trend of sterilisation rather than just cleaning with soap and water is a major trend why children are getting allergies to certain natural things. Their enviornments are so freakin sterile they don't build natural immunity to anything until it's too late.
As for washing after using the bathroom, of course, but I would much rather prefer a REAL soap (do they still sell that? I usually get special soaps from dermatologists) than that liquid crap they are putting in there...