I suspect the saying comes from a time more in the past when many many false relics would abound. Once the fervor around them died off, any ones deemed fake would disappear from circulation and the total amount would dwindle.