The number of slaves steadily decreased as the Rome stopped expanding. Additionally, by the time Rome turned into an empire, freeing of slaves became a show of magnanimousity and so frequent, Augustus (= around 0AD) had to put extra laws limiting it. Thus, the Roman empire had a shortage of slaves and couldn't just "count on the slaves" for everything.