If you want truly accurate results, you have to have a dedicated machine – not a VM (unless you’re controlling the hypervisor, and can guarantee no overcommit), not a container, but an entire machine – and tightly control every aspect of it. Updated packages? All results are invalid until you can prove the baseline hasn’t changed.

I don’t think most people need this level of accuracy, but it is something to consider.

Database queries are even worse, because not only do you need representative data at a similar scale (or you’ve scaled both hardware and working set accurately), but you have to hope that the table/column stats in prod and test are kept at a similar level of accuracy, which is more art than science.

Did the query regress because of the change, or because writes to an unrelated table have increased load and the analysis daemon is now struggling? Did it regress in test because someone else pushed a change to a rarely-used query, causing buffer churn when CI ran your job? Etc.

I agree with all of this. Because of this, even at massive scale it’s hard to justify spending the time and effort needed to get this level of testing.