You're free to use other compuational models in algorithm analysis, which define what costs you're assigning to which operations. If you use big-O on top, then the actual numbers won't matter, but which operations you count certainly do.

For example you could count only non-cached memory accesses on a particular model cpu. See "Idealized Cache Model" from https://en.m.wikipedia.org/wiki/Cache-oblivious_algorithm for example.

This is part of algorithms analysis that a lot of developers skip: these are just tools, there's many models available to use with them, or if you want to, just use wall-clock, with all of the pain that comes with that.