It's a tough one as there are good arguments both for and against it.
The original intent was to put people off from making meaningless short posts, just to say "something" (like "me too") and to allow people to voice or vent their agreement or displeasure about something without sticking their head above a parapet. It also helped to avoid some unpleasantries by allowing people to just put a thumbs-down without having to actually post, which I think helps avoid arguments continuing (at least a little).
It also helps show that just because someone may post something obnoxious, it doesn't mean that we as a community endorse the sentiment if there are a ton of negative votes.
But ...
Some people clearly misuse it and you see in the patterns that they go on a binge of up or downvoting everything certain people's posts, clearly doing so based on the person and not the content of the post (when they are innocuous). If the person doing this was visible, they may do it less. Of course the downside is you then get into arguments on why someone gave you whatever reaction.
Overall I think I'd favor having the voter be publicly visible, similar to facebook, but with a few more options:
Love / Like / Dislike / Angry
Laugh / Cry / Hug / Puzzled, maybe more (ideas?)
And of course:
Told-you-So Smug Grin (like your wife gives you when you finally have to admit she was right and you were wrong)