Tracking down numerical errors is brutal. I worked at a company re-implementing some MATLAB research code into optimized C++ (with parallelization where possible), where output had to match exactly bug-for-bug[1]. The algorithms were obviously very simple compared to weather forecasting on supercomputers, but it was still incredibly tedious dumping & comparing MATLAB debug values at dozens of points across multiple scenarios, each of which took hours to run. We even calculated one of the datasets they gave us would take 86 years(!!!) to process in the original MATLAB code. Sympathy for the team working on this. Fond memories of getting enormous arrays of NaNs after a fix.

[1] Bug-for-bug means bug-for-bug. There was a clear attempt to sort an array in the research code, which was implemented as a single pass through the data just deleting elements that were smaller than their direct predecessor. Research code is weird. It's one of the few places in software where "hack it till it works" actually produces valuable results.