Genesis 6:4 - The Nephilim were on the earth in those days—and also afterward—when the sons of God went to the daughters of humans and had children by them. They were the heroes of old, men of renown.
If we go only by the Genesis account, the Nephilim were 'heroes' and 'men of renown.' There are two other mentions, one in Ezekiel (which refers to mighty warriors, but does not use the word Nephilim) and one in Numbers (which, presumably, refers to their existence after the flood? The verse above seems to imply this as well).
The verse in Genesis seems like it was added in at some point, as it doesn't seem to fit in context. In any case, they seem like heroes of legend and not wicked dominators of men (though the verse in Numbers implies that they did terrorize people). Many cultures revere the powerful people who led them and/or kept them safe, and over time these characters are mythologized.