However, there are a few open source startups out there which started open from the beginning, such as Drupal or WordPress. You could also have a look at NewsBlur which got some traction after Google shut down Reader.

Whether or not being successful depends on a few things:

First, will open sourcing really make the product better? From my experience people tend to overestimate the capabilities of the open source community. This community does not translate to "legion of free developers at your disposal" but require an proactive engagement and community management. Are you well prepared for that?

Second, the nature of the product and the audience: If you're selling server software, for example, such like RedHat, there will be those "private" and "small" users which take your product and use it on their own. But there are also corporate users who will buy SLAs and first and foremost want someone to blame for things that are not working. Second, there are products who benefit from centralization. NewsBlur for example may less valuable for individuals if it would not provide a "hosted service". The same goes for Facebook. Their asset is more the user base than the technology (even if that's, of course, also a huge stack).

Third, what is your "open source strategy". There are companies like Pentaho or Alfresco which offer an "open core" and sell additional features for that. I would consider that more as closed than as open source, in some way it is a bit like a "demo version".

I would love to hear what you've build so far, because open source in a commercial environment is a topic of interest for me.