The problem is that you might find yourself at a point where it is just not all that enjoyable anymore. Your adviser stops caring about your research, or you are told that you need to get N papers published by the end of the year to stay in the program, etc. Maybe you find that you are unable to work with your adviser, and there is nobody else who will take you on as a student. I have seen a number of people leave grad school over such things.