The N64 only had 4 megs of unified RAM. 640x480x3bytesx2buffers would be 1800k. It was technically possible to run that without the expansion pack. But, losing that 1350k compared to 320x240 really hurt. 4x fillrate cost hurt even more.

N64 had a form of antialiasing support https://www.google.com/patents/US6999100 That was the motivation for the "9-bit RAM". Worked well enough for the time and was cheap enough to use. Much cheaper than running at a higher rez.