Even today people collect nice looking rocks, or minerals even. And I know that poor people would leave pebbles at altars when doing pilgrimage. So is it really so far fetched to say that these rocks were tactile enough for someone to keep them?