Isn't that largely offset by cultivating a lot of rice, that being so much more efficient as a crop?

What other things are at play, for example, in Japan's bonkers hight population across history?