The Smurfs were hated because of Gargamel the Wizard.
He was the bad guy even within the episodes, so I don't know if that reasoning makes sense.
All this despite the fact that, you know, this was a child's cartoon and clearly fiction.
I was actually allowed to watch the Smurfs because my parents truly saw through the superstition of some Witnesses, despite being hardliners in other respects.
And that's what it was, same with Harry Potter: superstition. A bunch of simple-minded superstitious people, afraid of their shadow and ungrounded in science. Figuratively throwing a pinch of salt over their shoulders when spooked by something.