Linus has said that the pain point actually starts around 1GB, especially when using a 3/1 userspace/kernel address split. At 1GB, you can't map the entirety of physical ram into kernel address space simultaneously, since you also have memory-mapped IO taking up space. Let alone mapping the same physical memory to multiple virtual addresses with different cache attributes, which is useful because approximately none of these SoCs have full cache coherence across all hardware blocks.