I suspect the convoluted arguments proposed for why the 13th Amendment must be unique are a direct consequence of the tortured logic needed to reconcile the institution of slavery with the principles on which the Constitution was based. Faced with a choice between acknowledging the government's role in legitimizing slavery or else arguing that it was merely a "private" matter which the government had thus far refrained from regulating they chose the narrative which minimized their share of the blame. This is akin to how the Interstate Commerce and General Welfare clauses have been perverted to justify all manner of legislation outside the narrowly defined scope of the federal government's enumerated powers.
You cannot pick and choose which principles you like. The Constitutional principles include the support of slavery. The Constitutional principles do not include universal suffrage. The Constitutional principles said nothing against expanding westward and stealing Native American lands. The Constitutional principles didn't prevent us from ruling the Philippines as a colonial power, or using government power to suppress labor rights.
To the contrary, I believe federal power was strengthened in part to be able to suppress the rights of the poor. The rebels of Shay's Rebellion fought against monetary policies and court practices which caused their lands and possessions to be confiscated. The elites and merchant class wanted to be able to call out federal troops should their own power be threatened.
What does "natural law" say about the right to confiscate land and possessions for failure to pay debts? For failure to pay taxes? What if those debts and taxes must be paid in hard currency, and that currency isn't available, and the government refuses to issue paper currency and allow it to be used as a substitute?
Quoting from the first paragraph of the ACLU link I gave: "With the notable exception of the Thirteenth Amendment’s ban on slavery, the individual liberties guaranteed by the United States Constitution protect against actions by government officials but not against actions by private persons or entities."
Quoting from https://academic.udayton.edu/race/06hrights/GeoRegions/North... : "the Thirteenth Amendment's prohibition against slavery and involuntary servitude encompasses both governmental and private action. Civil Rights Cases, 109 U.S. 3, 20 (1883). The U.S. Supreme Court has held that Congress may regulate private conduct under sec. 2 of the Thirteenth Amendment, which provides that "Congress shall have the power to enforce this article by appropriate legislation." Jones v. Alfred H. Mayer Co., 392 U.S. 409 (1968). Such power includes determining what constitutes the "badges and incidents of slavery and the authority to translate that determination into effective legislation."
Quoting from https://www.encyclopedia.com/politics/encyclopedias-almanacs... : "The Thirteenth Amendment is one of the few constitutional provisions that directly implicates private conduct. That amendment, the first of the three "civil war amendments," flatly bans slavery and involuntary servitude. It acts directly upon private entities; slaves were owned by private businesses and individuals. The Thirteenth Amendment, however, has not been interpreted to provide a significant source of constitutional proscription against acts of private discrimination."
Quoting from http://law2.umkc.edu/faculty/PROJECTS/FTRIALS/conlaw/thirtee... "The Thirteenth Amendment, unlike most provisions in the Constitution, is self-executing, in that it directly reaches-even without action by Congress- conduct by private individuals (slave holders)."