We give too much credit to that little book. Let's see if a got this right. According to the Tower of Babel story, the many human languages were created instantaneously by god; see Genesis 11:1 and Genesis 11:6-9, those verses states that there was just one language at the time. But if you examine Genesis 10:5, Genesis 10:20 and Genesis 10:31 you will find out that that actually the various languages evolved gradually over long periods of time.
Wait it get even better. God worries that people could build a tower high enough to reach him (them?) in heaven, and that by so doing they will become omnipotent, Genesis 11:4, 6. Well earth atmosphere is more than 62 miles thick.
We are not going to get into details about the extreme cold, wind or lack of oxygen at that altitudes and let’s say that we just go to 31 miles with is the altitude used by the weather balloons. Remember, airplanes usually fly at 6 miles (35,000 ft).
As of today the world's tallest building is Taipei101 located in Taipei, Taiwan and is 1,667 ft. height. Floor area is 4,440,100 sq ft. It even has a Tuned Mass Damper suspended from the 88th and 92nd floor to offset movements in the building caused by strong gusts. The building weights 700,000 tons.
The structure is made of steel and though steel had been produced by various inefficient methods long before the Renaissance; its use became more common after more efficient production methods were devised in the 17th century. Further refinements in the process, such as basic oxygen steelmaking, further lowered the cost of production while increasing the quality of the metal in the mid-19th century.
Even more, a pyramid's design, with the majority of the weight closer to the ground, means that less material higher up on the pyramid will be pushing down from above: this allowed early civilizations to create stable monumental structures. We can’t build a structure (building) more that 10 stories high without steel, not them, not no, it will collapse by its own weight.
I don't think that those people got the materials or technology to build something that big and tall. I do believe that it's a good story if you what to explain why there are so many languages around you and you don't have a comprehensible answer. It also help to feed that primal fear we have engrave deep in our DNA to fear, condemned and judge all those things that we don't know or understand.
Take care,
Ismael