Jamelle Bouie (NYT) is something of a constitutional scholar (albeit of the popular book based variety) and he would eviscerate your argument. Recent pieces by him in NYT explain specifically why Madison's ideas about factions are not ideas about political parties.

https://www.google.com/search?client=firefox-b-1-d&q=bouie+s...