You could drastically reduce --if not outright eliminate-- the IRS and still collect taxes. Just simplify the tax code.
Heck, you could probably collect more in taxes while lowering the tax burden for most taxpayers (and keeping the lowest rungs untaxed). You could also more easily make sure 'the rich/corporations pay their fair share.' But the tax code isn't designed for this. It's a massive, byzantine mess that provides lots of employment for the tax prep industry and far too much power for the IRS and whoever wants to misuse it.
No one ever made things more complex because they wanted people to understand it better, or because they wanted it to work more efficiently. Simplify the tax code, and you fix a lot of problems. Unfortunately, it doesn't seem that congress wants to fix that particular problem. If politicians do not want to provide the obvious fix for a glaring problem, then they don't see it as a problem. Which means they're either monumentally stupid or perfectly happy with a bad situation.