I'm pretty sure that would result in people coming sooner and leaving later, because they dont want to be seen as "that guy that abuses the system"

Same issue with unlimited vacation time.