Hypothetically that was true. Until those founders started engaging in actual politics and became rabidly partisan.

There was a brief period when the Federalists collapsed and US effectively became a single party state with the Democratic-Republicans controlling everything but that was decades after the constitution was signed.