The power disparity between the sovereign States was such that the Electoral College was a necessary compromise to get the small/weaker states to sign the Constitution.
Then, as now, folks were worried about 2 or 3 States basically deciding and carrying the entire election. The whole point was to mitigate that power gap between large and small.
Removal of the Electoral College would basically renege on the terms in which the Constitution was signed. I wouldn't blame smaller/weaker States for balking (and I live in one of the biggies!).
It's a voluntary association of sovereign States - it's not the mafia!