Aside from things like 18th-century manor houses - a tiny proportion of any country's population, and absolutely stuffed with working-class servants - the lack of motorized transit meant that upper-class neighbourhoods were usually right in the centre of the city, within walking distance of the poorer areas. You really just have to look at NYC or London to see that. 19th century mansions are not that far from the historically poor areas. And even those people who did own manor houses usually had a townhouse. It wasn't until trains, subways, and automobiles arrived in the mid-20th century that commuting between the business centre and exclusive suburbs became possible.
After one country took this attitude to unfortunate extremes a couple of generations ago - we have tried to all get along a bit better.
Walk through an old city. Even in the victorian era, where families were packed in single rooms with no running water, they were just a 5-10 minute walk from stately townhouses of the well-to-do.
Now, we feel the need to separate ourselves with berms and fences.