I strongly agree with your comment, with a single caveat:

> You wouldn't see this being a problem in careers like HR/Management/Sales. Because an individual can still be somehow productive without being in the zone.

I think it could actually be the other way around. That kind of work is very conductive to be "in the zone". In my few attempts at managing groups of people, I've learned that the work that's easiest to give you the state of flow is one where "next actions" are always obvious, and you get rapid feedback. Work where you can - and often should - substitute moving forward for thinking things through. Work where thinking gets distributed, outsourced to group interaction. That is sales and management, and I imagine lots of HR also.

So in a way, I think "people people" are "in the zone" much more often than we are, which is why they may not understand just how hard it is to get there when your tasks are brick walls requiring intense concentration.

You can totally crash manager flow.

I wouldn't recommend doing this on purpose, but if you say something that forces the whole room to stop and consider, you can often break the conversational flow and leave a meeting of managers dead in the water.

You can also do something that wipes an individual manager's mental cache and they'll spend days trying to remember what everyone is up to and what's happening.

I think your final sentence is close, but I'd rephrase it: for managers, those disruptions and conversations in your schedule are them dealing with things that require intense concentration. I think most of them understand what disruption feels like, they just don't associate sitting at the computer typing with that focus because it's not the mechanics of their role -- their focus manifests differently.

My managers only rudely interrupt me when they, themselves, are clearly still processing something their boss said to them and they need my input to unblock that process. Or some kind of thing I need to do now, again to unblock whatever they have going on.

I think you hit the nail on the head, I have definitely experienced a lot of this, even when dealing with managers I enjoy working with and that have an understanding of engineering-type workflows.

I'm not sure the metal state that you are referring to really makes a difference in terms of deadline estimation. The way I see it is that, in the grand scheme of things, if you are consistently missing the targets then either you get fired or, if the problem is systemic, the company folds.

In other words, if you are not getting fired and the company isn't folding, and that neither is happening anytime soon, then everything is just "business as usual" and missing a deadline here and there is just part of the natural statistical fluctuations that everyone have come to accept according to some unwritten rules.

I would go as far as drawing parallels between "being in the zone" for a software engineer and "meeting sales targets" for a salesperson. In that sense, not being in the zone every now and then is just as reasonable (or unreasonable, depending on your employer) as missing sales targets every now and then.

I would add that many "deadlines" are actually just arbitrary project management bullshit that exist to create a sense of "urgency" because the people that make the promises to customers don't trust the people that do the work.

But on an individual level, procrastination and slipping deadlines has a deep seated psychological basis. The best way an individual can deal with this problem is by addressing internal motivations and their own emotional state-- that's definitely NOT in the PM-BOK. (see https://www.nytimes.com/2019/03/25/smarter-living/why-you-pr...)

The problem is deadlines in and of themselves, it quite often leads to student syndrome[1].

I've seen it from the small up to the very big project. And I also believe it's also a little self feeding, after cramming so much for the first deadline you feel you deserve a little break, right?

We need to develop far more steady working practices harnessing intrinsic motivation rather than drive through fear derived motivation of deadlines and other anathemas of healthy work ethic.

[1] https://en.m.wikipedia.org/wiki/Student_syndrome

> in terms of deadline estimation

In my experience, deadlines are usually arbitrary to begin with. They are the result of annual/quarterly budget allocation much more than actual customer needs. Nothing of value is lost when they are missed.

I understand the need to divvy up budgets, especially in publicly listed companies. What strikes me as nonsensical is the particular inflexibility with these budgets and their poor correlation to actual customer needs. Perhaps this is merely a symptom of the modern business world, where "the business" refers to a class of people rather than the actual business.

I believe that OP is talking about estimating deadlines for yourself. If you estimate a deadline and then due to some reasons your mental state deteriorates, that makes it considerably harder to achieve it. Sales targets and deadlines provided externally would have very different connotations imo.

> if the problem is systemic, the company folds.

This is not the neccessary result. If a company is well established in its bullshit dealings it can survive for decades with inefficiencies and in some perverse cases even thrive.

> if the problem is systemic, the company folds.

Why do you think that? Imagine we have two similar companies, and one of them has mediocre deadline estimates, while the other would be mediocre except they also underestimate everything by 50%. Shouldn't both companies have similar success in the market?

> There is a huge productivity difference between people that wake up in the zone vs any other zone

There is my secret. I am never in the zone, and neither is anyone else in my company since they’re asked for status updates, meetings and ‘war rooms’ every few minutes.

I guess the one positive is that it makes estimation easier :D

I've only been coding professionally for a few years and I've too noticed dramatic productivity differences some days compared to others that aren't related to unexpected technical aspects ... it's just the volume, speed, accuracy of my coding all by itself it seems.

With something that takes a lot of focus I think mental state is absolutely huge.

We see athletes go through warmups and traditions before they start a game to get going not just physically but mentally ... makes me wonder if I should do the same.

Personally, it's much easier for me to be invested in a task when what I'm doing is 'interesting' to me.

Almost always and ironically, when something's interesting to me it's usually more difficult work and thus, contains far more unknowns, hence making estimates that much less reliable.

The solution for artists and writers is to work consistently. Obsessing about whether or not you're "in the zone" isn't productive. Instead, just get started, work every single day on your field. Seinfeld had a calendar and he had to write an X on every single day after he wrote jokes. I don't see why any high-functioning individual in any field couldn't learn something new every day.

This is also the solution for developers. You sit down and do the work. Some days are better than others. If you have rituals or strategies to maximize that productive time, that's great, but don't sit around waiting for the muse to inspire you.

I find it helpful to have:

- resumable tasks - I want to sit down and work, not sit down and figure out what to do

- multiple projects - if today isn't a great day for intellectual pursuits and I keep getting interrupted maybe I'll do an admin task or fix some simple bugs. Also works for when something is stalled.

- acknowledge you cannot control everything - if the progress is delayed in ways the estimate didn't anticipate, document them and keep working

> - resumable tasks - I want to sit down and work, not sit down and figure out what to do

I've found this indeed helps me get back to productivity. But at the same time, if I know what exactly needs to be done, it would be hard for myself to stop until it's finished. And that's when multi-day coding binge happens, followed by a few days of fatigue and low productivity. Kind a of a catch-22 for me...I wonder if there's anyway to be moderate about it.

I am not able to do that now. I have a family that needs to be there. When I was younger, sure, I'd pull all-nighters. But those days are gone. When I work now,I have to use my time wisely and be laser focused. I also found when I was working crazy hours, the quality of the code wasn't great. I learned to sleep on problems, then I'd solve them quickly first thing in the morning.

I'd say that not talking about context switches is weird.

Sometimes I'm jumping between 3 different projects and having talks with 2 people about $new_project within day and it feels like at least 1/3 of the day is wasted on context switch.

In my case, I'm a "wee bit on the spectrum," so to speak.

The minus is that I don't relate with people on the same level as most. It used to be fairly crippling, but I learned to compensate, and these days, it's just a bit annoying.

The plus is a "flow" that is incredibly deep and productive.

I long ago, gave up on "scientific" estimates. They make good catbox liner; not much else.

Breaking a project into layers and explicit "packages" (modules) helps. It allows me to evaluate complexity a bit better; but it always ends up being my "gut." I've been developing software for more than thirty years, so it isn't actually a bad metric. I am a bit of a cynic and a pessimist. That helps.

But the thing that works for me the best, is "JIT estimates." I won't (as in refuse to) give an estimate for the long term. If someone has a project plan in mind, I will try to plan a scope that I "feel" will be doable within that scope, then reduce it. If it is just me, I'll reduce it by about 25%. If it is a team, I'll reduce it by at least 50%.

I will then avoid specific estimates until I start on a package/layer. If I think it will take longer than I'd like, up front, I may reduce the scope. It's easier when it's a small[ish], discrete, atomic job, as opposed to a months-long aggregate effort.

That said, in my experience, the boardroom will always have a classic waterfall schedule, and they are the ones that sign the checks. They aren't always thrilled with my approach. That's why I have waited until I'm running the show to do it.

I'm working on a project now, that is fairly ambitious. It's the kind of thing that usually requires a team of five to ten engineers, and I'm doing it on my own. Thankfully, a really significant part (the backend) is done, tested, and released (for a couple of years). That puppy took seven months. I've been working on a native iOS frontend for about five months, but took a break for almost a month, as I needed to give a class.

In this effort, the folks I'm working with have no choice. They have to abide by my schedule, as I'm the only one doing the tech work. They are actually thrilled with the speed at which things are coming together. I credit my "always ship quality" approach for that. It almost eliminates "QC surprises," and gives unmatched transparency.

Indeed, this is a standard cause of procrastination: the belief that our future selves will be more in the "mood" to do the work, while our current selves aren't. Unfortunately, in the future we will still be our current selves.

So, similarly, we give estimations based on this rosy picture of our future selves, who will somehow work the rest of the week in unbroken eight-hour stretches of concentration and flow.

Huge thanks for talking about that. I thought I was the only one. I'm really unproductive when I have budget issues, can't stop thinking about it.

>And I've yet to see an SE or any other human that can wake up in the zone every single day. It's impossible.

I wake up in the zone every single time because I sleep what I need. I also do exercise and eat well. It is that simple.

In fact I wake up two times because I sleep in between hours of hard work. I copied that from tennis players and other elite sport players. Rafael Nadal plays a multiple hour match and goes to the Hotel to sleep(At 12 AM!!) after that.

I see no difference when the stress is mental hard work instead of physical. I am an elite player too.

I believe what you call "in the zone" is code for "having rested from an stressful situation" or not.

I see people trying to get by by using stimulants like coffee to work, but that is deluding yourself. It is just removing the signal from your body that you need to rest after stress.

You don't see this being a problem in HR/Management/Sales because the intensity of their work is usually very low. But those jobs could also become emotionally very stressful too (having to fire someone, pressure from above).

Most people can not even control or manage their own time and waste 2 hours commuting each day. Something has to give.

> I wake up in the zone every single time because I sleep what I need. I also do exercise and eat well. It is that simple.

Tried that, didn't work. Hell, at some point in my life, the only consistent way for me to get into the zone was to be sleep-deprived.

The zone isn't about stress per se. It's about focus. Low levels of stress can often help here (acting as rails that bounce you back onto the track if you start to veer off). High levels of stress will of course prevent the state of flow, but if you can kill off that signal somehow, you may have a chance to get back into the zone.

> I see people trying to get by by using stimulants like coffee to work, but that is deluding yourself. It is just removing the signal from your body that you need to rest after stress.

Thing is, people often have no other choice. The body doesn't have a "snooze" button - once it considers something an emergency, it doesn't understand the signal was received and noted, but there's nothing that can or should be done about it now. It'll just keep bugging you.

I recommend that you read about flow then.

The original idea of flow(in the zone) comes from this book and author, but is obsolete now:

https://www.amazon.com/Flow-Psychology-Experience-Perennial-...

I recommend this book and audiobook about gammification to understand flow better.

https://www.amazon.com/Reality-Broken-Games-Better-Change/dp...

Whatever breaks the game in your work, stops the flow. For example, instant feedback is very hard to get creating software unless you design for it(like using REPLs and reusing almost everything).

Other things that the book mentions break your game too(not having clear rules, being alone)...

Another important thing is thinking too much rationally. The logical mind is so slow getting results that you break the instant feedback.

Also working too much in low level has way slower feedback than working with higher level languages. I metaprogram lower level languages for this reason, I am orders of magnitude faster.

>The zone isn't about stress per se.

Sorry, I was not eloquent enough. when I talk about stress I am referring to stress in general: Mental, emotional, or physical stress.

That is, if you are doing something difficult mentally, you stress your brain.

For every stress you need time to recover.

You can do 10 hours without recovering if the stress is low. If the stress is high, you will only be able to do 4 hours.

It is very easy to understand, you can walk for 8 hours, but you can not run for 10 hours. A marathon is about 4 hours or less.

What I am telling you is that if you run for 4 hours hard mentally and do not recover, you will not be able to work another 4 hours of hard work, unless you rest.

Nothing will do, no technique will do unless you rest.

Of course you need to do right lots of other things too. But don't say it is impossible just because you can not do it.

ProTip: Human psychology and experience are sufficiently variable that One True Way responses for complex activities are exceedingly rarely universal.

What works for one person often does not for others. What works for a person at one stage of life often doesn't at others. What works in the short term often does not in the long.

Another anecdote. I'm someone who's made "sacrifices"[0], giving up alcohol, leave parties early to make sure I always go to bed at the same time (no exceptions), getting my diet in order, exercise etc etc. Literally prioritizing my sleep above everything.

I'm experiencing exactly what you describe. There's zero ramp-up time to get in the zone. I've gone from having 4 productive hours on a really good day, to having 8, at work, and several after work. I just get up in the morning (no alarm necessary), sit down with a clear mind, and start working.

I also do something similar to waking up twice. But it's more just lying down and staring at the ceiling for half an hour in the afternoon, if I feel it's necessary

[0]It's funny how the me of 3 years ago would consider these things huge sacrifices yet after I've done them it feels ridiculous to describe them as such

I've heard the theory that flow requires periods of struggling with learning or improving a skill, and that the intense learning is a precursor to having sufficient skill to be 'in the zone' relative to the challenge at hand. Does that jive with your observations? Is learning just another challenge that you have sufficient skill for, or do you find that cycle isn't necessary, or something else?

I am sure many people are like me in that there are constant meetings, questions and other interruptions to the point that you never actually get into the "zone".

I have adjusted estimates saying that exact thing to a publisher, but never in a traditional software environment unless it was a physical injury or sickness.

Regarding mental state: this study covers June 22, 2020 to the "present" (the paper is dated April 1, 2021, but I don't think it's an April Fool's joke). I suspect that people are to some extent calibrated to pre-pandemic expectations. I know I can't get things done like I could have in the Before Times, either because of mental state or because of more distractions at home than usual.

Figure 1(b) of the paper might suggest that people's estimates are worse on tasks that require flow, although I don't think there's enough data to be sure.

Energy.

I’ve seen a couple ugly conversations where someone didn’t understand why an engineer couldn’t do 2 tasks that take half a day/week/sprint in the same time interval, and it comes down to both of those tasks being very high maintenance, or requiring a high degree of concentration. It may only take 20 hours to complete but it sucks up 30 hours worth of energy. You need something brainless for the other 20 hours, not a second task of that ilk.

Overriding priorities has nothing to do with "bad mental state". It only has to do with changed priorities and scope.

Missed deadlines have to do with need to fix other more urgent stuff.