N64 had a form of antialiasing support https://www.google.com/patents/US6999100 That was the motivation for the "9-bit RAM". Worked well enough for the time and was cheap enough to use. Much cheaper than running at a higher rez.
> most games implemented their own pseudo z-buffer in microcode.
Just two or three IIRC. Mainly Factor 5 games.
Goldeneye use Z-Buffer, bullet texture overlaps because the poly offset value used to apply bullet texture on walls is always the same.
> The general theme of the technical analysis was that the N64 had a lot of "features" that had a massive performance impact to use, so many of them went unused.
I wouldn't say that.
1) The N64 is a complex console. It still inherit the "all electric" tradition of 16/32 bits consoles but focusing on 3d computation and shading so it need skilled peoples and time to do interesting things with it (in its last days, N64 had very impressive games).
2) With that, Nintendo devs provides GBI (graphic binary interface) which was not such efficient on the vector computation side (but generic enough to allow peoples to use it).
3) N64 had a all-wired per pixel processing unite (a GPU named RDP), meaning it could provide high quality pixels like never before but this was costly.
4) Because texture had to be stored physically "inside" the RDP chip for efficient processing (remember, 3DFX didn't even exists at this time and the GPU word was a concept), memory cost (and, I suppose, size limitation) were important so they choosed to focus on pixel quality over texture definition (there is actually many hack to "visually" improve texture definition: http://imgur.com/2EeH5Yg).
In the case of Mario 64 though, many of the models and polygons didn't have textures but used super-simple single-color flat shading. So texture fill rate wouldn't not have been a constraint.
Most games also didn't use a custom microcode, but used one of a handful of microcode libraries for the RCP that Nintendo provided. A notable exception to this is Factor 5, which did write their custom microcode to improve graphics performance, with great results.
"The Z-buffer could not be used because it alone consumed the already constrained texture fill rate."
-- https://en.wikipedia.org/wiki/Nintendo_64_programming_charac...
I vaguely remember a technical analysis on the N64 that explained using the Z-Buffer had such a dramatic negative impact on performance, that only a very small handful of games actually used it. The general theme of the technical analysis was that the N64 had a lot of "features" that had a massive performance impact to use, so many of them went unused.
If I remember correctly -- most games implemented their own pseudo z-buffer in microcode, which is why a lot of games on the N64 have issues with textures popping through each other incorrectly -- Goldeneye's bullet hole textures stick out in my mind as a good example of this.
However, I'm trying to find the analysis, and I'm not having any luck. Maybe I'm just remembering things incorrectly.