I think we need to look at the verses again, the Lord here is not talking about two individuals or groups, but one.
The interlinear translates it "and if THAT servant should say evily in his heart." This tells us that the once "faithful servant" become evil when he felt the "Lord delayeth."
Quite interesting, if we take the Society's position that the slave is a CLASS, then Judge Rutherford is a dead ringer for the "evil Servant." How is one faithful then evil? It is all wrapped up with the second advent. Russell taught that Jesus' parousia was in 1874, Rutherford also taught this. Eventually Rutherford stated "the Lord delayeth" and moved the date up to 1914, forty years later, he also "smote" his brethren [the schism of 1917] and began drinking with the drunkards, it is a known fact Rutherford was a drunk.
"People in glass houses, shouldn't throw stones"