Weren't N64 games over all higher resolution than PS1? I remember PS1 being just a jagged mess compared to N64

I think the main difference was textures: on the 64 they ended up being blurrier/flatter and therefore less pixelated. That said, there were definitely PS1 games that looked great and weren't cursed with jaggies (Wipeout, for instance).

I'm not sure about resolution, but the N64 had built in AA and AF.

Maybe I'm super confused here, but I thought there was no such thing as "higher resolution" when it came to analog TV signals. This was before HDMI/HDTV. Your resolution was whatever NTSC/PAL dictated. Or am I wrong?

The screen resolution is the same but the resolution of the image can change. If a system's output renders to the entire screen then just imagine a 1 pixel output would make the entire screen the same color. 4 pixels could have 4 quadrants of colors. 16x8 pixels would have 128.... etc. The max resolution would be the max resolution of the display. Though a topic of interest may be sub-pixel rendering where a system/font/game can take into account the RGB color pixel layouts in an LCD screen into account in order to improve the visual quality of the max resolution.

So a system can have double the resolution of another and games can even choose to render at different resolutions depending on how intensive it is. Some games will choose lower resolution but more polygons and some might choose higher resolution with fewer polygons, among other choices. There are even modern cases where console games will change the resolution they run at to keep framerates solid https://www.halowaypoint.com/en-us/forums/6e35355aecdf4fd0ac...