Very interesting. I've also had similar ambitions. I found this essay inspiring (although pretty long):

http://www.holisticpolitics.org/NewParty/NewPartyManual.php

It basically outlines strategies for how a new political party can realistically gain traction. As you allude to, an important factor is focusing on elections that can be won rather than just running to make a statement.

Perhaps it's possible to write software to analyze past election results, polling, etc. to find races where a third party has a chance and would be the best to allocate resources to, kind of like a Moneyball of politics.