We are still at a point where minor changes can result in dozens of "change detected" triggers, of which we are not sure how to determine whether they represent corrections or regressions (it's usually not at all obvious.)
I've also been weary of building a large "accuracy testing suite" for the SNES, on the fear that if I come to the wrong conclusion on something, others will emulate that inaccuracy and won't question it because of deference to my code.
Tests also have the unfortunate side-effect of being gamified (example: http://tasvideos.org/EmulatorResources/NESAccuracyTests.html), which is a very bad thing. Failing a test because we don't know the answer is infinitely better than passing the test using am unverified guess. But when people reduce a comprehensive and complex set of tests into a percentage score, you get a very skewed picture of what actually matters. (think of how browsers jumped through hoops to pass the Acid CSS tests back in the day.)
It is however something that's important, and hopefully I'll have the time one day to work on this more.
Then you can form a reference set of captures, real hardware and the emulator. Screen to screen emulation with diffs could then be an accurate tool to measure against actual hardware and the accuracy gap between the two. No? Controlling the random number generator sounds like it might be key. And some player traces might be really fragile in the input timing search space.
Dolphin (the GameCube/Wii emulator) already does this, actually. They make dumps of various states of the graphics pipeline across multiple revisions of the source and generate image-based diffs of the results.
https://fifoci.dolphin-emu.org/about/
Here's an example.
Also, none of that automation you just did will work w/the actual hardware.
An infamous bug of that variety exists in the SNES game "Speedy Gonzales - Los Gatos Bandidos", where one specific element in one level would freeze the game on most emulators due to a really subtle hardware emulation issue.
https://mgba.io/2017/07/31/holy-grail-bugs-2/#speedy-gonzale...
> Also, none of that automation you just did will work w/the actual hardware.
You might be surprised. Consider what the TASBot team has done with the NES, SNES, and various other consoles -- http://tasvideos.org/TASBot.html
Not to mention only a very small subset of SNES TASes are done on cycle-accurate emulators.
Not only is there unitialized RAM and I/O registers, and some analog effects, the really big elephant in the room is that the system has two oscillators. A ~21MHz CPU/PPU crystal oscillator, and a ~24MHz SMP/DSP ceramic oscillator.
Given that not only do these exact frequencies change between systems due to margins of error on clocks, they also change slightly as the system runs (and gets warmer, for example.)
Every SNES game has sound routines that synchronize the CPU to the SMP.
It wouldn't be possible to make a literal 1:1 play log unless you a) ran a custom register and memory initialization at system startup, and b) replaced the two oscillators with a much faster single oscillator and then used a clock dividier to drive both the CPU and APU off of it.
(You can TAS certain SNES games anyway, of course. It really depends on how the game is programmed to react when the exact CPU<>SMP communications change. If it seeds a random number generator based around the PPU H/V counters that are polled after a CPU<>SMP sync for example, forget about it.)
Maybe it'd be possible to modify some boards to use a CPLD (or an FPGA) for synthesizing those clocks from a common source. This could eliminate most (all?) uncertainty.
I wonder how much of a lift it would be to take a bunch of TASes and turn them into regression tests...
To get that, you would need a set of hardware debuggers plugged into the bus and chips. And a lot of inside knowledge to decide if a deviance is random enough to not have to be emulated.
Last time I had to get SEM micrographs in resolutions between 2K and 10K, we paid 475€ for 2 hours of work.
While I wouldn't want to pay this out of my pocket and gift it to the world, I don't see how it's less feasible to make better die scans than to build the testing equipment (PPU breakoutboard) needed for automated testing.
Especially if the current owner of the already decapped dies were willing to lend them to someone else willing to do the footwork. I think it should be easy enough to find 19 people willing to donate 25€ for this kinda effort.
On the other hand, the dies would have to be plasma coated with Gold/Platinum, which might be bad, I'm not an expert. So, perhaps you were refering to this when you said higher-res scans are less feasable than making automatic testing happen. ¯\_(ツ)_/¯
> We would have to retest the entire SNES library to be sure such a change did not cause any regressions, which just isn't practical with a library of 3,500+ game titles.
Running automated regression-tests against a library of known-accurate titles actually sounds extremely practical. This sounds like standard software-engineering practice to me.
I'm surprised that a test-harness like this doesn't already exist by now given the relatively mature state of SNES emulation. It seems like it would be an extremely helpful tool to benchmark accuracy of new or existing emulators, prevent regressions from advanced optimizations, etc.
Automated testing could transform the 'blind guessing' approach on tweaking PPU timings from a game of 'whack a mole' to a structured, iterative linear algebra solver. Which could end up being quite a bit more feasible than acquiring 100x die scans of the chip and tracing the logic from the hardware directly.