> but even during the civil unrest periods things were OK if you stayed in your villa with your slaves and vineyards

Well, I mean, this is if you happened to be very rich. Also, of course, for quite a bit of the period it put you in the firing line for proscription, and other Rich Roman Hazards. Particularly after the end of the Republic, the Roman Empire tended to be a fairly hazardous place for rich people; odds were good that you'd end up, at least passively, on the Wrong Side at some point.