Well, the standard in computer science is to optimize algorithms and discuss their performance for the worst case, so if we assume he's talking from a CS perspective, I find it natural to assume he's talking about that. There are a few exceptions, like QuickSort, but if some tells me to optimize some algorithm, I'll assume he's asking for optimization of the worst case unless he says otherwise or it's clear that the worse case is much rarer than some other case (I'd ask if I think that).

These considerations certainly exist in every day code as well. If you change the daily reporting process to have a worst case run time of 25 hours then you lose, even if the mean run time is cut in half.