The biggest limiting factor is the ability to etch deep holes with a high aspect ratio. Shrinking individual memory cells along any axis is of limited use (fewer electrons means less durability), so you can't make layers much thinner. Making holes wider can get you more layers, but isn't really a net win for total density. So far, nobody has started stacking more than two decks of layers, so I don't know if we have good data on how cost and yield scale when going all-out on string stacking to reach extreme layer counts. As an alternative, there's R&D into using wider holes, and then splitting the vertical channels in half, giving you two semicircular memory cells per layer rather than one circular cell.

There's also a need to keep shrinking the peripheral circuitry as the number of memory cells stacked above each mm^2 of buffers and charge pumps keeps growing.