If the precise values of these floats is important in your domain (which it very well may be), a combination of approaches would probably be good! Would love to hear how well this approach works for you guys. Keep me updated :)
Obviously any time you're working with floating-point sample data the precise values of floats will almost always not be bit-accurate against what your model predicts (sometimes even if that model is a previous run of the same system with the same inputs as in this case); it's about defining an acceptable deviation. I guess what I'm saying is that for audio software, a peak-peak error of 0.1 equates to a signal at -20 dBFS (ref [email protected]) (which of course is quite a large amount of error for an audio signal), so perhaps using higher-resolution graphs would be a good idea.
(Has anyone made a tool to diff sixels yet? /s)
For this, just thinking about sound, I wonder if you could invert the reference wave form and add it to the test to see how well it cancels? Then instead of just knowing there was a diff, you could get measurements of the degree of the diff.
4.9600227 0.094451904297 -0.014831542969
4.9600454 0.089172363281 -0.0092468261719
4.960068 0.087493896484 -0.0065612792969
4.9600907 0.090179443359 -0.0028686523438
4.9601134 0.093963623047 0.0060729980469
4.9601361 0.095367431641 0.020538330078
4.9601587 0.094299316406 0.035186767578
4.9601814 0.09228515625 0.045013427734
4.9602041 0.089691162109 0.051422119141
4.9602268 0.086059570312 0.058929443359
Columsn are: [time in seconds] [left channel sample] [right channel sample]This was generated using
sox somefile.wav somefile.dat
You can reverse that by reversing the argument order above.- The quantization of the graphs is a feature to add some tolerance to the tests. I admit this is a mixed blessing.
- This is a lot more opaque to someone looking at a text file of the test output than what is described in the post.
I agree though that you probably want to augment this with some form of assertion about noise level to check the high frequency smaller components.
--pb, CTO of Fourier Audio Ltd.
(e.g. if you somehow managed to introduce a constant DC-offset of +0.05, with the shown step size of 0.2, these tests would probably never pick it up, modulo rounding.)
That said, these tests are great for asserting that specific functionality does broadly what it says on the tin, and making it easy to understand why not if they fail. We'll likely start using this technique at Fourier Audio (shameless plug) as a more observable functionality smoke test to augment finer-grained analytic tests that assert properties of the output waveform samples directly.