News for hackers. Stuff that matters.
Re:The Edge of Emulation by wgjordan on Tuesday January 21, 2020 @11:50PM (#22114739)
> If we latch too late, after cycle 40, Full Throttle Racing will glitch. But if we guess a value too soon, say on cycle 0, that might be too early, and it might cause issues in another game.

> We would have to retest the entire SNES library to be sure such a change did not cause any regressions, which just isn't practical with a library of 3,500+ game titles.

Running automated regression-tests against a library of known-accurate titles actually sounds extremely practical. This sounds like standard software-engineering practice to me.

I'm surprised that a test-harness like this doesn't already exist by now given the relatively mature state of SNES emulation. It seems like it would be an extremely helpful tool to benchmark accuracy of new or existing emulators, prevent regressions from advanced optimizations, etc.

Automated testing could transform the 'blind guessing' approach on tweaking PPU timings from a game of 'whack a mole' to a structured, iterative linear algebra solver. Which could end up being quite a bit more feasible than acquiring 100x die scans of the chip and tracing the logic from the hardware directly.

Re:The Edge of Emulation by near on Wednesday January 22, 2020 @03:22AM (#22115668)
Screwtape built exactly such a regression-testing harness for bsnes already.

We are still at a point where minor changes can result in dozens of "change detected" triggers, of which we are not sure how to determine whether they represent corrections or regressions (it's usually not at all obvious.)

I've also been weary of building a large "accuracy testing suite" for the SNES, on the fear that if I come to the wrong conclusion on something, others will emulate that inaccuracy and won't question it because of deference to my code.

Tests also have the unfortunate side-effect of being gamified (example: http://tasvideos.org/EmulatorResources/NESAccuracyTests.html), which is a very bad thing. Failing a test because we don't know the answer is infinitely better than passing the test using am unverified guess. But when people reduce a comprehensive and complex set of tests into a percentage score, you get a very skewed picture of what actually matters. (think of how browsers jumped through hoops to pass the Acid CSS tests back in the day.)

It is however something that's important, and hopefully I'll have the time one day to work on this more.

Re:The Edge of Emulation by sitkack on Wednesday January 22, 2020 @12:56PM (#22120085)
I don't know enough about SNES hardware or the level of in system debug, but could one generate a hardware test jig that was able to record player traces that could be re-run in the emulator and have the screen grabs compared? Given the system state, random number generator and the series of up/down/a/b one should be able to recreate the same game evolution and video output from the emulator?

Then you can form a reference set of captures, real hardware and the emulator. Screen to screen emulation with diffs could then be an accurate tool to measure against actual hardware and the accuracy gap between the two. No? Controlling the random number generator sounds like it might be key. And some player traces might be really fragile in the input timing search space.

Re:The Edge of Emulation by nonbirithm on Wednesday January 22, 2020 @02:52AM (#22115519)
> Running automated regression-tests against a library of known-accurate titles actually sounds extremely practical.

Dolphin (the GameCube/Wii emulator) already does this, actually. They make dumps of various states of the graphics pipeline across multiple revisions of the source and generate image-based diffs of the results.

https://fifoci.dolphin-emu.org/about/

Here's an example.

https://fifoci.dolphin-emu.org/compare/5108854-5102201/

Re:The Edge of Emulation by kevingadd on Wednesday January 22, 2020 @01:02AM (#22115020)
You can't just start it up and run it for 5 seconds, though. A given game might be a hundred hours long and only exercise certain functionality at minutes 5, 37 and 60. It's non-trivial to automate the playback of the whole package and then verify the whole stream against a known good. So you'd need to identify the key points in each game that are important, and replay those with save states and hope that the save states aren't unintentionally encoding corrupted data that came from an existing bug in the emulator you used to run them.

Also, none of that automation you just did will work w/the actual hardware.

Re:The Edge of Emulation by duskwuff on Wednesday January 22, 2020 @03:31AM (#22115710)
> A given game might be a hundred hours long and only exercise certain functionality at minutes 5, 37 and 60.

An infamous bug of that variety exists in the SNES game "Speedy Gonzales - Los Gatos Bandidos", where one specific element in one level would freeze the game on most emulators due to a really subtle hardware emulation issue.

https://mgba.io/2017/07/31/holy-grail-bugs-2/#speedy-gonzale...

> Also, none of that automation you just did will work w/the actual hardware.

You might be surprised. Consider what the TASBot team has done with the NES, SNES, and various other consoles -- http://tasvideos.org/TASBot.html

Re:The Edge of Emulation by menmob on Wednesday January 22, 2020 @07:33AM (#22116950)
In no way will TASBot ever be automated. It can require hours (and sometimes dozens of hours) to get a single game to sync on real hardware, and that’s if a cycle-accurate emulator was used. If not, it’s near impossible to playback a TAS.

Not to mention only a very small subset of SNES TASes are done on cycle-accurate emulators.

Re:The Edge of Emulation by theresistor on Wednesday January 22, 2020 @01:54AM (#22115227)
The input stream for an SNES isn't very complicated - just the state of a few buttons at every cycle, and the transition edges are quite sparse relative to the clock frequency. You could easily record human play sessions and replay them.
Re:The Edge of Emulation by near on Wednesday January 22, 2020 @03:30AM (#22115703)
The SNES output is non-deterministic across runs.

Not only is there unitialized RAM and I/O registers, and some analog effects, the really big elephant in the room is that the system has two oscillators. A ~21MHz CPU/PPU crystal oscillator, and a ~24MHz SMP/DSP ceramic oscillator.

Given that not only do these exact frequencies change between systems due to margins of error on clocks, they also change slightly as the system runs (and gets warmer, for example.)

Every SNES game has sound routines that synchronize the CPU to the SMP.

It wouldn't be possible to make a literal 1:1 play log unless you a) ran a custom register and memory initialization at system startup, and b) replaced the two oscillators with a much faster single oscillator and then used a clock dividier to drive both the CPU and APU off of it.

(You can TAS certain SNES games anyway, of course. It really depends on how the game is programmed to react when the exact CPU<>SMP communications change. If it seeds a random number generator based around the PPU H/V counters that are polled after a CPU<>SMP sync for example, forget about it.)

Re:The Edge of Emulation by vardump on Wednesday January 22, 2020 @06:57AM (#22116755)
> Given that not only do these exact frequencies change between systems due to margins of error on clocks, they also change slightly as the system runs (and gets warmer, for example.)

Maybe it'd be possible to modify some boards to use a CPLD (or an FPGA) for synthesizing those clocks from a common source. This could eliminate most (all?) uncertainty.

Re:The Edge of Emulation by near on Wednesday January 22, 2020 @11:49AM (#22119181)
It would certainly help a lot. People have talked about doing this, but I don't think there's been enough interest to actually make it happen.
Re:The Edge of Emulation by _qwfv on Wednesday January 22, 2020 @02:12AM (#22115324)
There's already a community of people that does this called tool assisted speed runs (or TAS). These players already record their inputs for replay and are comfortable tuning them to consistently hit edge cases.

I wonder how much of a lift it would be to take a bunch of TASes and turn them into regression tests...

Re:The Edge of Emulation by AstralStorm on Wednesday January 22, 2020 @03:30AM (#22115706)
The input steam is half the story, you should also match the whole observable state of the machine for an accurate emulation, because some other test might actually depend on it. Triggered on all the clocks and decide which clock matters every case by hand.

To get that, you would need a set of hardware debuggers plugged into the bus and chips. And a lot of inside knowledge to decide if a deviance is random enough to not have to be emulated.

Re:The Edge of Emulation by xenadu02 on Tuesday January 21, 2020 @11:58PM (#22114763)
I don't know if SNES dev kits still exist but if they do they might prove useful. Such kits sometimes have additional debugging facilities because they're also used for validation (or they share some elements). Not sure if anyone was doing that sort of thing as early as SNES though.
Re:The Edge of Emulation by mlatu on Friday January 24, 2020 @06:12AM (#22137111)
> more feasible than acquiring 100x die scans of the chip

Last time I had to get SEM micrographs in resolutions between 2K and 10K, we paid 475€ for 2 hours of work.

While I wouldn't want to pay this out of my pocket and gift it to the world, I don't see how it's less feasible to make better die scans than to build the testing equipment (PPU breakoutboard) needed for automated testing.

Especially if the current owner of the already decapped dies were willing to lend them to someone else willing to do the footwork. I think it should be easy enough to find 19 people willing to donate 25€ for this kinda effort.

On the other hand, the dies would have to be plasma coated with Gold/Platinum, which might be bad, I'm not an expert. So, perhaps you were refering to this when you said higher-res scans are less feasable than making automatic testing happen. ¯\_(ツ)_/¯