That's just because we need to define the set more gracefully: "n! is the product of the first n natural numbers".

The negative factorial issue arises from a slightly awkward specification of the set, not from jameshart's core idea.