I'm not sure I agree here, because the manager has the ability to impact all those other hours.

A good manager helps remove the hurdles that will make those 40+ hours immensely more productive. A bad manager adds to those hurdles, potentially whittling productive hours to nothing.