No, middle-class people had servants back then. It wasn't just the rich.

The lower class, of course, largely were servants.

The industrial revolution opened up other avenues for employment.

Edit: see, for example, http://www.bl.uk/victorian-britain/articles/the-victorian-mi...

"A middle class woman was entirely dependent on a supply of domestic servants, ranging from the untrained ‘slavey’ to a staff of several highly trained specialists."