It is difficult to overstate the increase in opportunity over the last hundred years for women in particular. Back then, there were so very few avenues to the middle class for even an extraordinary single woman. Today, Mary Poppins would probably go to a fine college and be able to land a job as a pharmaceutical sales representative at the very least.
The point is, for a large class of servants, you need both immense wealth disparity and a large class of insanely rich people. We've increased a) and in the process, b) has gotten smaller.
Don't get me wrong, many people are rich - but only a few are rich enough that they can afford servants. This might change if the job situation gets worse, increasing the disparity so that more rich people now qualify as insanely rich.