According to both ancient and current theology and religious history, the "tree of knowledge" was not considered literal until American Fundamentalism was born during the era of the great revivals in New England that gave birth to religious groups like the Pentecostals, the SDA, the LDS, and of course the JWs.
The narrative of the "forbidden fruit" comes from the second of three creations narratives. The first in Genesis 1, states that all fruit from all trees and plants are given for food to humanity. But in the second, more ancient, the story of the tree of knowledge is introduced as an alternative or further explanation of the first mythology. (The third creation narrative is the Noachian flood, based on the more universal belief that the physical universe is built up of elements of a previous universe and that water is the basis for all material life.)
In Judaism the account is generally viewed as a primitive explanation that describes how humans get along with the Creator, a give-and-take that results in the creation of the nation of "Israel" (which literally means "nation of contenders with G-d"), named after the patriarch Jacob who, legend has it, continued the "wrestling" match with YHWH begun by Adam and Eve.
In early Christianity and current Catholicism/Orthodoxy, the narrative of Genesis 2 is the basis for the doctrine of "Original Sin." Interestingly it is not viewed as historical but written employing narrative devices that tell not so much the origin of sin as it does the fact that humanity has been marked by the nature of concupiscence since its very beginning.
The eras of American history known as the First and Second "Great Awakenings" gave birth to a form of Sola Scriptura that denied thousands of years of Biblical scholarship, both Christian and Jewish. Developing in what would be termed the "Burned-over district," these groups would invent a form of literalism never seen before on the world religious schema.
The idea that the "tree of knowledge" was somehow real lead to the belief that the taking of the "forbidden fruit" was a historical explanation for Original Sin. This idea is shared by religions from the Burned-over district, including the Millerites, the LDS, SDA, and JWs.
Due to having practically no exposure to the original theology of the traditions that developed the narratives, many who have left these groups still consider the American-invented religious views as representative of the actual genre employed in Genesis 2. This has led to many criticizing the original text when in reality it was never accepted as historical in the first place. Any "errors" found in the text are compounded by the fact that some, after leaving these groups, have never considered that the narratives never attempted to be historically or scientifically accurate in the first place. The blame for this comes, of course, not upon any ex-members but upon the systems which abused the narratives in the first place.