There is no clear answer to this question. We can only guess as to the origins of this tradition.
1. An early Midrash Lekah Tov (also known as Pesikta Zutra) 35:20 relates that each of Jacob's sons took a stone and put it on Rachel's grave to make up Rachel's tomb. Here and elsewhere we learn that by placingstones on the grave one participates in building the tombstone. We do not find any direct connection with our present practice, but we might ask if this is an ancient memory of this tradition.
2. We learn in theTosefot to Tractate Sanhedrin 47b (in the Babylonian Talmud), that Rabeinu Tam interpreted the "golel"; as a large stone slab that they place on the grave as a marker and this is called the tombstone as is written (in Genesis 29:35) "It is the tombstone of Rachel's grave."
Rabeinu Tam, the grandson of Rashi, goes on to explain that there were smaller stones that were set under the sides of the large stone that rests on them so that it will not bear down too heavily on the deceased. These smaller stones are called the "dofek" (upright stones upon which the large stone rests). To these smaller stones it was a custom to attach a marker until a large slab is found, lest the place of the grave be lost.
3. Rabbi Solomon B. Freehof quotes the book Ta-amey Ha-Minagimim (The Reasons for the Customs, pp. 470-471) of late nineteenth century author, Ithak Sperliing: "We put grass and pebbles on the grave to show that the visitor was at the grave. It was a sort of calling card to tell the deceased that you have paid him a visit." (See also Orah Haim 224:8). Furthermore, we find in the Shulhan Arukh Yoreh Deah 376:4 : Now it is practiced after the grave is covered to pluck up grass or pick up a stone and put them on the gravestone, which is only for the honor of the deceased that the grave was visited. (See Freehof, Reform Responsa for our Time, 1977, pp. 291-293).
A contemporary respondent, Rabbi Ovadia Yosef, confirms this custom, relying on Eliyahu Rabba 224:7 as his source (Responsa YabiaOmer IV,Yoreh Deah 35).
In former days one did not mark a grave with marble or granite with a fancy inscription, but one made a cairn of stones over it. Each mourner coming and adding a stone was effectively taking part in the Mitzvah of matzevah ("setting a stone") as well as or instead of levayat ha-meyt ("accompany the dead"). Of course, the dead were often buried where they had fallen, before urbanization and specialization of planning-use demanded formal cemeteries. Nowadays one can no longer bury a relative in the back garden, or on their farm, nor may a deceased traveler be interred by the roadside.
Therefore in our day one tends to stick a pebble on top of the tombstone as a relic of this ancient custom, and it is still clear that the more stones a grave has, the more the deceased is being visited and is therefore being honored. Each small pebble adds to the cairn - a nice moral message. This has become slightly spoiled by the cemetery authorities clearing accumulated pebbles off when they wash down the gravestones and cut the grass.
There is a custom of plucking grass from the cemetery. The Chief Rabbi of Safed, Moshe ben Yosef Trani, (1500- 1580) stated that we pluck the grass after the burial as a reminder of the resurrection. The idea stems from Psalm 72:18: They shall spring up as the grass of the field.
If you prefer, there is a psychoanalytic explanation of this custom. Dr.Theodor Reik explains:
People in different parts of the world believed that the soul of the deceased haunts the grave for a long time. On account of that continued fear, prehistoric men rolled great rocks in front of to graves, so that boulders should prevent the dead from escaping and plaguing living relatives.
If so, what is the role of the small stones?
Reik elucidates: The many small pebbles form a substitute for the one big boulder. It is as though the survivor who had visited the grave of a relative and so exhibits his piety to the dead, protects himself from their envy or hostility by putting those stones in their abode, preventing the dead from escaping. (Theodor Reik, Pagan Rites in Judaism, 1964, pp. 44, 48.)
My colleague, Rabbi Andrew Straus offers the following explanation:
Ritual is a way of expressing our emotions and spiritual needs. We need physical acts to express these things for us, to make them concrete.
Placing a stone on a grave does just that. It works in several ways:
1) It is a sign to others who come to the grave when I am not there that they and I are not the only ones who remember. The stones I see on the grave when I come are a reminder to me that others have come to visit the grave. My loved one is remembered by many others and his/her life continues to have an impact on others, even if I do not see them.
2) When I pick up the stone it sends a message to me. I can still feel my loved one. I can still touch and be touched by him/her. I can still feel the impact that has been made on my life. Their life, love, teachings, values, and morals still make an impression on me. When I put the stone down, it is a reminder to me that I can no longer take this person with me physically. I can only take him/her with me in my heart and my mind and the actions I do because he/she taught me to do them. Their values, morals, ideals live on and continue to impress me - just as the stone has made an impression on my hands - so too their life has made an impression on me that continues.