http://www.bible.ca/islam/library/Gilchrist/temple.htm
..........
Thirdly, secular history in no way supports the Qur'an's claim that the Ka'aba was ever a place of monotheistic, non-idolatrous worship. The first mention of the Ka'aba is found in the writings of Diodorus Siculus who, about 60 BC, described it as a "temple greatly revered by the Arabs". Accordingly the Ka'aba dates back at least to before the time of Christ. But this fact only helps to support the final conclusion we shall draw in this chapter. It certainly does not in any way suggest that the Ka'aba existed before the Jewish Temple. On the contrary, before the time of Muhammad, the Ka'aba was only known as the principal shrine of pagan idolatry of the Arab world in and around Mecca.
We do have clear evidence, however, that the Ka'aba is not of monotheistic origin. We refer to the black stone built into its east corner known as al-hajarul-aswad. Before Muhammad's time the Arabs worshiped stones and the black stone was one of these objects of worship. Not only was the kissing of this stone incorporated into Islam, but the whole form of the Hajj Pilgrimage today is fundamentally that of the Arabs before Islam. Muhammad only changed the meaning of the formalities - he made no attempt to change the forms and rites of the pilgrimage themselves.
Some have suggested that stone-worship among the Arabs arose out of veneration of the black stone, but this is highly improbable. Any form of veneration of a dead stone - especially to the extent of bowing down and kissing the stone - can only be identified with pagan idolatry rather than pure monotheistic worship. Even Umar was reluctant to imitate the pagan Arabs by kissing the stone and only did so because he saw Muhammad do it. But in our view Muhammad likewise was only perpetuating one of the forms of Meccan idolatry and we cannot possibly see how veneration of a form of idol-worship can be reconciled with the worship of the one true God.
Secular history knows of only one form of pre-Islamic veneration of the Ka'aba and that is the idolatry of the pagan Arabs. There is no corroborative evidence whatsoever for the Qur'an's claim that the Ka'aba was initially a house of monotheistic worship. Instead there certainly is evidence as far back as history can trace the origins and worship of the Ka'aba that it was thoroughly pagan and idolatrous in content and emphasis. Certainly in the six hundred-odd years between the destruction of the Temple and the final conquest of Mecca the Ka'aba was purely a shrine of thriving pagan idolatry. Therefore the Ka'aba cannot have become the form and place of true worship in God's providence when the Temple of the Jews was destroyed.
http://pnews.org/PhpWiki/index.php/IslamsPaganRites
............................
STONE WORSHIP, THE KA'ABA, AND PILGRIMAGE
The Ka'aba was the sanctuary in Mecca where the different pre-Islamic deities were worshiped.Among the images of the various gods and goddesses, the image of Allah was preserved alongside those of the female deities mentioned above. The sanctuary of Mecca, like any other house of god among the Semitic peoples (e.g., Beth-el), was in a rectangular shape and for this reason the Arabs called it Ka'aba, which is an Arabic word meaning "cube." Some scholars maintain that the word means house and they have suggested that it comes from an Ethiopic word meaning double or two-storied building." But this does not seem to be correct, as the Arabic meaning of Ka'aba very clearly signifies rectangular.
The different idols were placed either around the rectangular structure in the open air or inside a niche (qibla) under the vault. It is believed by some circles that the rectangular structure was probably constructed after the shape of the typical nomad's tents. The Ka'aba dates back to the second century c.E.; thus, contrary to the text of the Koran and Muslim exegetes, it could not have been made by Abraham. The Old Testament makes no mention of the patriarch traveling to Arabia, much less building a sanctuary in Mecca! More importantly, there is no chronological or archeological evidence to prove that Mecca is older than the first century C.E., and Abraham, if he really existed, probably lived around 1800 B.C.E. The story was apparently fabricated by Muhammad himself to attract the attention of Jewish Arabs in the early days of his mission to win converts.
In the pre-Islamic period stones, especially soft and cube-shaped ones, were often considered sacred by desert Arabs and were venerated as the residences of gods. Nomads living in cube-shaped tents themselves probably thought the gods, too, would have similar abodes but of stone.
Among the many places of worship in Arabia, the Ka'aba was respected more than other sanctuaries because it held the Black Stone, which the pre-Islamic Arabs believed was given to them by the deities of the skies to be worshiped. The Black Stone is thought to consist of hardened lava, or basalt, but it is not easy to determine its real nature as it has been touched and kissed rapturously by millions of people since it was placed in the structure. It may have a meteoric origin. Islam has thus perpetuated the ancient pagan rite of stone worshiping by ordaining the pilgrimage to the Ka'aba as the fifth pillar of the religion.
Another practice in this festival was the ceremonial abstinences. The pilgrims would stay away from profanity, irreverence, and sexual intercourse, since each of these practices is prohibited in the presence of the gods of the pantheon.
The most diverse pagan tribes of the Arabian Peninsula would take part in the festivities of the pilgrimage. Since the performing of ceremonies was possible only when there was peace in the land, and as the Arabs were constantly fighting one another, the carrying of arms in Mecca during the pilgrimage was forbidden.
All of these ancient pagan rites survive in Islam today. Each year Muslims from all over the world make pilgrimages to Mecca during the Arabian month of Dhu- Hajja, and they perform the same rituals that prevailed several centuries before Islam.
There are many websites with info on the "holy" stone worshipped by Muslims. Just google and you will get info.