The author of the linked post has a entire book about the legal cases that set down these categories in the US. He does seem to think they are both arbitrary and bad (and somewhat unique to the US, unlike slavery and racism which are not a uniquely US phenomenon), in roughly the same way that a libertarian would object to everyone being pigeonholed as either a republican or a democrat since they believe themselves to be neither.