This part of Harari's book have been debunked quite a few times already: his views of pre-agricultural are romanticized and doesn't match with the work of the specialists of this period. (While Harari is indeed an historian, he's a specialist of medieval times and has little authority on early-humans history).

Btw, I'm not even disagreing with you about agriculture, I just wanted to point out that quoting Harari on that has really little value.

That said, I'm no historian, just a layperson who found Harari's work (so far -- I haven't yet finished "Sapiens") thought-provoking and interesting. In the relevant early chapters his perspective is refreshingly different from the norm -- in some ways comparable (for me) to Zinn's "A People's History of the US", in that it provides a PoV sufficiently removed from the standard narrative to serve as a reminder of how shallow and incomplete any one-sided version of events must be. The GP's citing of Pinker likely belongs in this camp, too: referencing an author whose ideas have merit (eg Pinker's computational theory in "How the Mind Works"), independent of their ultimate status as authoritative works.

You might think this, but the historical pattern -- everywhere -- is that farmers stay where they are and get conquered by more mobile non-farmers. They don't so much fight as submit.

What happened was that farming was so successful that the human population boomed. When famine did occur, it caused more deaths because there were more humans.

Also it’s a bit of a trap: as soon as you switch over you cannot go back, as foraging cannot sustain as many people. You will have to constantly expand to feed the growing population.