The traditional 4.2BSD-style quotas (1983!) on linux also support quotas on unix groups. Not sure if you had this in mind, but anyway.

I suppose project quotas as outlined here would allow multi-group support though.

Another option could be sparsely-provisioned COW LVM volumes.