All is fun and games until you face and evetually get wiped out by more advanced farmer-based societies, with specialized labor (army), more developed tools (weapons and cavalry) and much more sophisticated social structure (elites deciding where to expand). Jared Diamond wrote a lot on this subject.