> the "winning" strategy is to have high-level scripting languages where you can ignore most of the details, which call into hyper-optimized, high performance libraries. For instance, when you're using Scipy, you call into Fortran and C almost interchangeably.

That gets you about 70% of the way there, but the remaining 30% is tricky. Unfortunately, the library-call stage doesn't easily (with generality) let you fuse operations. For example, consider the example of a finite-volume code, where the flux is some nonlinear function of the interface value, taken to be the average of left and right cells:

  flux = nonlinear_flux_op(f[:-1] + f[1:]) # Compute interface fluxes
  f[1:-1] += dt/dx*(flux[:-1] - flux[1:]) # Update cell volumes based on flux differences
  # Do something special for the boundaries

Each of the substeps here is fast. Sub-indexing these 1D arrays is trivial, and their local operations are highly vectorized. The nonlinear flux operator operates element-wise, so it's easy to compile it into a version that properly (CPU) vectorizes (and/or it might already be provided, like np.exp or *2). This code is also very readable.

However, real performance would require fusing operations. There's no need to compute a full temporary array for the interface values, but that's exactly what would happen in naive NumPy code. Likewise, there's no need to compute all of the fluxes before taking their differences.

A hand-optimized code would perform the full computation in a single scan of the array, keeping all of the data in CPU cache (blocking by a cache line or so in order to use vector registers effectively). Even if given the code naively written, an optimizing compiler would try loop fusion and get most of the way to the hand-optimized result.

Code such as this is relatively FLOPS-light compared to the amount of memory accessed, so effective use of cache can lead to an order of magnitude speed-up.

With a fair amount of work you can still achieve this result in Python, but to do so you need to use one of the optimizing Python compilation tools like numba. Unfortunately, this is not easy for the not-a-programmer domain scientist, since working with numba means understanding the intersection of Python and numba's imposed compilation rules.