You can trace the evolution of "LiveActor" all the way through until it ends up in Super Mario Odyssey.
Sunshine - https://github.com/shibbo/Corona/blob/master/include/actor/T...
Galaxy 1 - https://github.com/shibbo/Petari/blob/master/include/Actor/L...
Odyssey - https://github.com/shibbo/OdysseyReversed/blob/master/includ...
This architecture was so successful it ended up as the basis for all new Nintendo game development, so Breath of the Wild, Pikmin 3 and Splatoon, Mario Maker all use this new "Actor Library", or "al".
I have not looked at NSMBU, but NSMBWii uses a different core structure originally developed (as far as I know) by the Zelda team. I think it's mostly phased out these days, as is the set of "egg" libraries developed by the Mario Kart: Wii and Wii Sports teams.
I mean, you're right, it's not a literal implementation of OAM in the sense of controlling the same things OAM controls. I was speaking kinda metaphorically.
NES/SNES OAM was useful for reading back entity physics data (because it gave objects X/Y position registers) which meant that developers (incl. Nintendo themselves) often chose to rely on the OAM-object "components" of a entity as the canonical handle for tracking the entity in the game physics (Rather than having a table somewhere in work-RAM of separate "physical" components for entities.) Games like SMW literally just index a table of actor behaviors off the OAM-object's name-table data; what an entity "is" from the game's perspective, is determined by what it currently looks like!
Since the OAM had a finite size, this reliance on OAM for tracking entities forced games into a structure where entities' lifetimes are coupled to the lifetime of their OAM-object representations. Which meant that every NES/SNES game relying on OAM to track entities needed an algorithm for dynamically allocating OAM-object slots to entities; and so, for evicting entities if OAM was exhausted. (Level design was done with a hard eye for avoiding OAM "thrashing" by keeping entities spaced apart, but the system still needed to be able to handle the case where mobile entities ended up following you and piling up.) Which brought into existence the common OAM LRU cache-eviction algorithm—i.e., the practice of "despawning" the oldest off-screen entities when new on-screen entities need OAM slots.
This determined a lot about the design of these NES/SNES games. It made mobs in these games into things that would lose their state whenever they were scrolled "far enough" off the screen; which in turn forced a design where—rather than a level just running a "start script" that would spawn entities at initial positions, tracking them in RAM from then on—you instead had adopt a hybrid approach where entities had both an OAM-object representation, and also an associated "spawner" (usually existing just as static level-data in ROM, though sometimes coupled to a bitflag tracking destroyed spawns) that would trigger [re]spawning for the entity.
SM64 is essentially "emulating OAM" in the sense that it assigns entities handles in a fixed-sized buffer, and then uses a very OAM-like logic (basically, "memory pressure" on this buffer) to decide when entities should be de-spawned; and then uses spawners to recreate entities that have been de-spawned due to this memory pressure (meaning that most entities don't "exist" until you get close enough to them.)
SM64 didn't need to do things this way; the N64 has enough RAM to track all the entities in every SM64 map at once, IIRC. They chose to impose this constraint artificially, in order to continue to build SM64 levels according to the design philosophy they had "discovered" due to the original constraints of the OAM system.
Later games in the Mario series, if-and-when they choose to have this de-spawn/re-spawn tracking feature†, are essentially "pretending to have OAM", but not really emulating it the way SM64 does. For example, Mario Maker de-spawns entities when they're scrolled sufficiently far off the screen, in a way that mimics OAM sufficiently well that re-spawning and enemy spawner semantics still work—but which isn't really an OAM-like system, in that there's no static buffer with memory-pressure causing de-spawning (and in fact, as long as the entities are willing to squeeze into one visual screen, existing entities will never be forced to de-spawn.)
† You could get a very interesting analysis of the way Nintendo probably internally divides/project-manages the Mario games, by just determining which titles "emulate" OAM the way SM64 does; which titles loosely mimic OAM, like Mario Maker; and which titles don't even bother with de-spawn/re-spawn tracking at all, but instead have persistent physical entities that just "go quiescent" when they're out of sight. (IIRC there's no Mario title that uses the fourth option—pure view-frustum culling of distant models that continue to "tick" while culled.)
I don't know how the SNES worked, but AFAIK most NES games did not track objects in this way. Instead, the game engine maintained its own buffers containing object state and copied necessary information to OAM every frame.
OAM only stored graphics state for the rendering hardware, which is not a convenient form for the game engine for a number of reasons. For instance, objects are nearly always composed of several OAM sprites placed next to each other, objects that are not visible during a given frame are not present in OAM, and a single animated object can switch between so many different graphical forms that it would be complicated to identify which object corresponds to a graphics tile from OAM. Additionally, OAM doesn't have extra room for non-graphical object state (like behavior timers or velocity information).
Nintendo is quite clearly second-to-none on the design/creative end, how much does that translate to the technical aspect of game development? Speaking purely in terms of software.
I find this particularly interesting in the context of a company that appears to retain many of the same programmers today as they did 30 years ago, when software development was obviously much different.
(Side-note: I've always wondered how the mini-games in IS's WarioWare series work—whether each game is entirely custom code, or whether they've come up with some sort of DSL for specifying reflex games. If the latter, I would bet that that has a decent genealogy too.)
[0]https://en.wikipedia.org/wiki/Mario_Artist#Polygon_Studio
FTFY. I don't think the RPG elements of SPM should be ignored; the game plays very differently to any of the other Mario platformers.
It may not be to everyone's tastes but to simplify the matter for the sake of a quick jab is hugely unfair, especially given it has one of the most touching stories in the Paper Mario canon.
Picture a background jobs system like Sidekik/Resque. Imagine that one worker-node of this jobs system had a fixed-size ring array of jobs it had taken. Now imagine that you could push new jobs onto a specific node. And now imagine that the worker-node responded by not just overwriting one of the filled slots of the local jobs set, but actually ACKing said job to drop it from the global job-queue system. It's destroying a real entity with persistent global identity, in order to reclaim the slot that the local representation of that entity takes up.
That's what OAM is, when combined with the design pattern I'm talking about. It's a ridiculous system that'd never fly in a business; but it happens to work for games, where you control the world such that you can make the world hold "reminders" for the state you destroyed.
"Miyamoto: We were using the Super Mario 64 engine for Zelda, but we had to make so many modifications to it that it's a different engine now. What we have now is a very good engine, and I think we can use it for future games if we can come up with a very good concept. It took three or so years to make Zelda, and about half the time was spent on making the engine. We definitely want to make use of this engine again."[1]
[1] https://web.archive.org/web/20040619165414/http://www.miyamo...
One thing I know (and can be seen in this repo) is that SM64 emulates a version of the NES/SNES "Object Attribute Memory", as a pure-software ring-buffer. (I'd love to know whether that carries on to later titles like Galaxy, 3D World, NSMB(U), Mario Maker, etc.)