Beyond that, categorizing anything as "inherently legitimate" is fraught with peril. One could perhaps surmise that all things are legitimate within a democracy, but so long as the favors of the majority come at the expense of the minority, that claim is disputable.
Beyond that, I don't really know that Rousseau's contract would invalidate a North Korean government as it exists, and North Korean citizenship precludes the option of defection.
The biggest flaw I see with the assertion that "if you want to live here, you agree to pay taxes" is that there's no opt-out policy in America as it is. If you object to the taxation, and wish to opt out, your only path is expatriation, with its associated expatriation taxes.
And yes "inherently legitimate" was a poor phrase to choose.
And I believe many founders leaned heavily on Rousseau's work. Hamilton and Madison particularly in the Federalist Papers.
And which is inherently fallacious, even going against Rousseau. Slavery as an institution was systemically agreed upon by the institution. There is no inherent legitimacy in violating the liberties of others, and while I agree that any such comparison is flawed, agreeing on slavery as an institution is no more or less legitimate due to populist demand than the insistence of taking some people's money and giving it to others.
Are there benefits? Undoubtedly; but even something mathematically proven as "good" (as if we could do such a thing) is not necessarily legitimate, however well founded the idea or the intent. That said, this is all philosophical, to be sure, but since we're waxing, may as well get fully waxed. (Don't ask me what that means, cause I have no idea.)
> And I believe many founders leaned heavily on Rousseau's work
No argument there.
Constitution, Article 1, Section 2, and Section 3, in their first paragraphs clearly define the legislative branch as a representative democracy.