If you do not get a Fellowship, you are expected to perform 20 hours worth of work per week (which you actually receive credit for), in addition to your time spent on classes and on your own independent research.
In other words, grad school is more like a low-paying job than a cost. You could make the case that you could be working a higher-paying job in that time, but then you are missing out on the flexibility of being a Ph.D., which allows the possibility of being a professor or getting hired for a pretty high income. Additionally, you have the opportunity to leave your mark on the field with your research.
Even more interesting is that Ph.D. work could lead you to your startup idea, as you unveil a new use of existing technology or a new approach altogether which leads to a money-making possibility. Furthermore, if you are exceptionally driven, you may be able to get some work done on your startup idea while attending grad school, because at least so far, I have found grad school to be significantly easier than my undergrad was, because I am taking less "real" classes.
It does not depend on being in grad school or not whether you can do what you want, it depends on having the money - either to pay for grad school, or for toying around with your startup.
I realize I made some oversimplifications, but still...