You are retracing the arguments made by Rawls in Theory of Justice. The result he proposes is a maximin principle that gives (grossly simplified) based on the maximum of access to social goods that is compatible with similar access for everyone else (equality principle). Differences are to be favored if they advance the situation of the least well off and if they are based on contribution/merit (differences principle).

Most commonly, maximin is extended to leximin i. e. Looking at the next least welloff. There are traditions of critique both from communitarianism (Sandel et al.) and libertarianism (Nozick et al.), both worth looking into.