The root cause of the development bandwidth problem was that we wasted vast amounts of time and energy on building misfeatures. And we built so many misfeatures because we had way too much development capacity in relation to our product planning capacity. So the BAs didn't have enough time to fully develop ideas before sending them to engineering, and were instead stuck throwing things over the wall when they were still half baked. All these half-baked ideas took time to build, and then they took time to fix after they bombed on the market, and they slowed everything else down since every new feature is something you have to ensure still works when you make a change in that part of the software. Day in, day out, we were taking on technical debt to cover our design debt bills.
So then we tried Agile, and man oh man did that make things worse. 'Cuz now instead of just lobbing things over the wall, the BAs had to be actively involved in every sprint. There goes any opportunity to sit down and concentrate, because now you've got a nattering horde of 20 developers all trying to get clarification on last week's decisions, which leaves less time to make this week's decisions, which means next sprint things will be even more sketchy and ill-defined.
The correct solution would have been to hire more business analysts, and they key thing we needed from those business analysts wasn't instructions on what to build. It was instructions on what _not_ to build.
I know one project in particular we did a release about every 7 months. Every release cycle they estimated 2 weeks for requirements, every cycle took a month, and yet the next cycle you'd see a 2 week estimate on the roadmap.
As for switching to Agile, I know some people would tell you that Kanban would identify this problem for you by around week 8. Did you guys ever try having development liaisons help them draft requirements that were precise from an engineering perspective? Sometimes that initial meeting works better when there are only three people in the room, instead of a bunch of exasperated people being condescending to you again while you're trying to do a good job. adjusts collar nervously I'm uh... asking for a friend.
Why 50% of a release's entire timeline? Because you're going to have overruns, errors, re-dos, and any number of random unplanned elements creep into the scope of your release. And the reality (I may get blasted for this blasphemy) scope creep is fine. As long as it pertains to the originally scoped requirements and is clearly adding value. "A delayed game is eventually good, but a rushed game is forever bad.", Shigeru Miyamoto. The important part of that quote is "delayed", you still need to make sure you release otherwise you're just spinning and wasting company time and money.
And whatever you do, don't try and "fill in weeks". If the release is ahead of schedule, let it be. Let it finish weeks early if it has to. Then, rinse and repeat. Admittedly this type of release process can be hard to sell because there's usually a negative emotional response to the idea of only scoping 50% of the total allowed release cycle and that somehow the other 50% is waste. It isn't, it's there to guarantee that the items scoped initially actually get done on time. That way QA can get to them in a reasonable amount of time, give feedback, and (during the second half of the release) engineering can fix the issues raised by QA. Business development can also rest comfortably knowing that the features scoped into the first half of the release are actually going to come out on time.
I know it sounds like "You're just adding slack." But it's intended to be more than that. If you see teams attempting to squeeze more items into the 50% release you know there's a prioritization problem within those teams. Which can be remedied by either: 1) reducing the overall size of the release cycle further (eg 8 to 4 weeks with 2 weeks as the 50% mark) to provide more flexibility into what goes into the product in a shorter time frame. 2) Helping those teams clarify what the high-level product direction and priorities should be.
Hopefully this 15 minute rambling comment is valuable to you. If not, I apologize.
[0]The mistake I made and the advice I can personally give, don't focus on up front high-accuracy estimates and designs. Come up with a reasonable design, your best guess estimate, and then implement as furiously as possible (coding best practices should still be enforced). You'll know much earlier in the release cycle what the real timeline for your feature will be than if you'd tried to spend all that time up front.
You don't mess with Brook's law!
I have yet to find a better text on how to properly manage software projects than, "The Principles of Product Development Flow: Second Generation Lean Product Development"[0].
[0] https://www.amazon.com/Principles-Product-Development-Flow-G...
This is 100% true and unfortunately gets distorted from the financial backers who pump them up for unicorn status. Quite a dichotomy:
"Software is eating the world!"
"The hard thing isn’t setting up an organizational chart. The hard thing is getting people to communicate within the organization that you just designed."
Not every project or human endeavor that involved wheels, fire, written language, plumbing, electricity, or radio ate the world, but every project that did not involve them eventually got eaten, for the simple reason that you needed to keep up with technology as a minimum. But beyond that, projects succeed or fail for the same reason they always have: understanding the actual problem you're trying to solve.
Uber's success is not due to software. Uber's success is due to its understanding of the transportation market and how to play the regulatory game to its advantage while never getting permanently in trouble. Uber's ability to deliver on the huge market opportunity they saw—i.e., its lack of failure—is due to software. We quite reasonably call this "success" when talking about technology, but if we're talking about companies succeeding or failing, that's not a discussion about technology.
So why not look at failure then?
Success has the false-positive issue. Failures have the false-negative issue.
Cargo-culting false-positives is going to be less harmful than cargo-culting false-negatives. (X Company didn't use stack-ranking and then failed. Clearly stack-ranking is bad for companies)
Have you ever met a billionaire who said, "I was lucky" and left it at that?
AKA in predicting overall success would a team with low tech expertise, but high sales expertise be have a better chance?
I expected him to go off and actually attempt building it, that way he would get some more experience.
And he did! He went off that weekend and wrote a proof of concept route planner, it was great to see him learn something new and complex
I have seen multi million dollar projects hit the same snags, horribly under scoped all because Mr Easy didn't think about asking all of was needed. They saw the tree and missed the forest
What's the expression? "Plan your system for 100x growth, but build your system for 10x growth". Something like that except for functionality rather than scalability.
That and supporting the system. Junior engineers tend to think about building the tech, which is honestly the easy part. Keeping it working until the last of your customers are finally ready to upgrade 10 years later is harder than junior engineers tend to think.
I know it's not always easy to do this, but that really is part of the challenge. There's still too many deathmarch projects lead by people trying to do things 'the right way'. If you're working on a new project for the love of science, please make sure you're releasing something valuable every single week if not every one or two days!
I think a big part of the disconnect being discussed originates in the myth that CRUD-based apps are fundamentally easy. This simply isn't true. The challenges of building a modern, complex, scalable CRUD app with good UX and a maintainable code base are not 'less than' challenges rooted in algorithms or bit bashing. Doing it well enough and fast enough to be competitive in the market is HARD.
Uber and Lyft are not just taxi apps. They are massively polished. When I use them I am astounded at how good they've gotten the UX and then I shudder at how hard that must have been across two mobile platforms and in the presence of very unreliable networks. I've ordered an Uber with no cell Internet using barely usable coffee shop wifi from a block away.
Then you have scaling, which brings up or creates issues you just do not see at smaller scales. Big O type considerations and weird edge cases that only come up once every billion transactions start mattering.
Then there is all the big data stuff. These services do tons of route optimization and analytics.
So yeah marketing, sales, fund raising, management, etc. also matter but tech often matters as much or more. It's just that junior engineers usually only see the tops of icebergs. This is why despite decades of pure biz people saying "tech doesn't matter" you still rarely see founding teams succeed without at least some tech founders but you do occasionally see even lone tech founders make it. (But those tech founders do have to know or be willing to learn the non tech stuff!)
> Internalizing that has been difficult for me as an engineer.
After I got over that particular hurdle, I would still think:
Ok, so "build it and they will come" is not a thing, got it. But still, engineering is most important by far. Without sales, support, marketing, or legal, you still have a product, just one that is adopted slower. Without engineering you don't even have a product to sell in the first place.
And that's true. But in the end it's just splitting hairs. It doesn't matter. Over the longer term, no successful, scalable business is ever going to exist solely as an engineering team. Every job function has a purpose[0] and gets you where you want to go faster; in some cases, lack of certain non-engineering job functions will make it impossible to even do certain things.
So sure, you can bootstrap a product with just a few engineers and nothing else, but unless you're ok with it being a niche-market product that you spend all your waking hours working on, an engineering-only org is never going to cut it.
[0] With the obvious exception of those people hired to satisfy the latest management fad du jour, or to give the CEO's deadbeat nephew something to do with his time.
Execs love to think this but it's just not true. Top heavy companies waste money. It's true that tech of not the only ingredient but for a tech company it's critical. Good tech is not a commodity. Top-tier engineers are worth as much as a good CEO. That is if you want to remain lean.
I suspect this is because everyone wants to not only feel as part of something but they want to make an important contribution. I venture to guess accountants at Google do not feel they make a difference to search. The culmination of parts is what makes the company whole and all those parts are required.
Its hard to accept you are not as important as you wish you were. Its a great step in the maturation process of one's professional career though.
Have you considered broadening your personal definition of technology past computing and engineering?
From Wikipedia:
> Technology can be the knowledge of techniques, processes, etc. or it can be embedded in machines, computers, devices and factories, which can be operated by individuals without detailed knowledge of the workings of such things.
Technology is the creation of systems. Those systems run as much on the users of that technology, and the members of the organisation supporting it, as they do inside of microprocessors.
Technology exists largely to accelerate the pace of development and of course, it's never a sole driver of any business. There is no reason why you can't create a small-scale uber with call centres and excel spreadsheets but I doubt it would reach the scale and impact of what Uber is today. Granted that, it's hundred times easier to figure out to technology than business, I won't trivialise role of engineering because it's something allows a business to near its global maximum.
The feeling that the CEO I was working with was doing mundane tasks was astonishingly - he even said that he could to his whole job using his smartphone.
But after trying to do the business stuff myself I realized that no one in all of this process is irreplaceable - finding a good software developer may not be easy, but it's also not easy to find a good CEO. This is important to realize that software developers shouldn't see themselves as some type of elite.
The main problem is that junior devs don't realize that 90% of business is based on networking and being friends with people who are in middle management and want to increase their reputation in their big corp. You don't get big clients because you're a genius hacker, most of the time those B2B applications are simple CRUD apps. You won't get that $500k because you simply didn't play golf with the key partners. I'm not even exaggerating, I've seen that the best deals are made in ones "spare time".
Maybe startups are another breed than the software companies I know, but don't drink the cool aid and don't assume that technology is your main differentiator. It's working with people all along.
I think it isn't even about scaling high-performance systems, it even starts with the mindset of those who think that software developers are entitled in some way, usually junior devs and people who never did any real business work.
Most of the unicorns are founded with engineering talent coming straight from Stanford, MIT, Harvard, UCB, Waterloo etc. you name it.
Making a company successful is as much about economics of scale than raw engineering and of course, both are closely related in this day and age. Engineering shapes what and how your products feels like in real-time (= understand very short code-to-production lifecycle).
It might not be politically correct to say but I am not here to make you feel better. Anyone who has actually been in a team at a company that has exponentially grown will tell you the same: your chances to ever survive that "first wave" of users is directly proportional to your ability to hire the best, which of course is a lot easier when they are your current or former classmates.
That's not to say there aren't companies founded by people not going to the schools mentioned above or that engineering talent is only available at these places. It is obviously not the case.
However, the density of talent at top-schools is absurdly high. And this is a game changer for most early stage companies and even more so for soon-to-be-unicorns.
But aside from knowing talented people, it takes visionaries to generate fantastic outcomes. Walt Disney for example stopped drawing cartoons himself early on because he had better artists around him. However, he had the vision that ultimately brought the best out of each artist.
The success of many companies, and probably all of the unicorns, has nothing to do with technology. The tech is necessary, of course, but so are desks and an accounting department.
Internalizing that has been difficult for me as an engineer.