The real problem was different. The PS1 cache was a real cache, managed transparently by the hardware. Textures could take the full 1 MB of VRAM (minus the framebuffer, of course).

In contrast, the N64 had a 4 kB texture RAM. That's it, all your textures for the current mesh had to fit in just 4 kB. If you wanted bigger textures, you had to come up with all sort of programming tricks.