Docker does allow for daemonless execution, but as you say one daemon per user will add a bit of overhead.

There's some tradeoff I guess though, between rootful setup and per user, as images duplication per user could add up.