The push towards more grains and carbs in the 70s, along with the war on cholesterol, helped usher in the age of type-2 diabetes. The continued demonization of meats and fats don't help, either.
As for energy-in/energy-out, I agree that it's too simple a model. I think we should be looking at what works for us and what doesn't. People lose weight and build healthy bodies on a seemingly endless array of diets and dietary approaches. So it seems as if they all work. I would say, find the commonalities, and also find what works for you. A diet that you cannot follow won't work, obviously.
Most dietary approaches will warn against processed foods and especially against foods with lots of sugar or starch. I think that's a good start. I've found that for me, it's not only sugar, but chocolate (specifically, milk-chocolate) that appears to have additional negative effects for me. So I've been cutting that out lately and my skin has been in much better shape than it has in some time.
Our bodies can adapt to a lot of approaches, but I think we each have our own specific needs. I am focused less on calories and more on finding ways to eat that work for me and in developing regular exercise habits. Try lots of stuff. Keep what works, discard what doesn't. Above all, don't punish yourself- your brain won't put up with that for long, and it's one reason that so many approaches fail.