This is the fallacy of long hours. You spend so many hours at work every day and every once in a while you have some late night inspiration or collaborative breakthrough or take advantage of some other opportunity, and then like any gambling addict you associate working long hours with the rewards. You ignore all the costs (stress, family strain, sleep deprivation, lower productivity, higher defect rates, etc.) It's classic confirmation bias.

Add to that the machismo notions that working long hours is working "harder" than otherwise and that working fewer hours shows less commitment and less strength. In reality it takes more focus, more smarts, and more discipline to work shorter hours effectively (sure, you can slack off by working fewer hours, but you can also slack off by working more hours too)