A lot of problems in software arise from choosing the easy route (i.e. gem install hairball), with no regards for the complexity it could bring into your project. Do this enough times and your project could easily end up like one of the projects described here: a massive mess of complexity that's excruciatingly difficult and risky to change and improve.
Re:Overcomplexifying, Underdelivering
by sjbase on Wednesday October 21, 2015 @02:00PM
(#10427229)
I worked at a major consultancy and this is definitely true. The major parties in a large enterprise IT project are (in shit-rolls-downhill order):
1: The enterprise itself
2: Strategic advisors (consultancy)
3: Integrators (often the same company as #2, or a competitor)
4: Software vendors (note the plural)
5: Third-party QA
All but #1 have a structural incentive to make projects bigger the same way a mechanic has a structural incentive to "find new problems." The client always has final say, but people underestimate how hard it is to keep saying "no" to multiple reputable firms all warning you against going too small.
That said, this kind of swindling is far from ubiquitous. There's a philosophy that blindly upping scope is shortsighted, and you do better in the long run by actually representing your client's best interest. Where I worked, I found it pretty easy to surround myself with people who thought this way.
Re:Overcomplexifying, Underdelivering
by ep103 on Wednesday October 21, 2015 @02:30PM
(#10427487)
Don't forget that many enterprises end up replacing their own staff completely with people on the consultancy payroll. Apparently it can make the company look better on wall street's books? So the end result is that 1 and 2 become the same people. ISPs are rife with this.
Re:Overcomplexifying, Underdelivering
by joshuarrrr on Wednesday October 21, 2015 @01:49PM
(#10427117)
Hi, I'm one of the authors of the piece. As my co-author would say, the industry is fueled by failed projects. But because outright failure is common, huge overruns can be seen as a success when they actually manage to deliver anything of use.
Re:Overcomplexifying, Underdelivering
by perlgeek on Wednesday October 21, 2015 @04:11PM
(#10428308)
> One solution is to try to make more realistic initial estimates.
But more realistic initial estimates won't win the contract for your company.
I don't know about the legal situation in the US, but in the EU, the lowest bidder gets the contract. So everybody has the best incentives of making a (sometimes even ridiculously low) initial estimate, and then find reasons to charge more.
And such reasons are always to be found, since requirements won't remain stable over the course of a multi-year projects. They never, ever do. And any change can be sold as an expensive scope creep that costs a few million USD.
Re:Overcomplexifying, Underdelivering
by haddr on Wednesday October 21, 2015 @12:04PM
(#10426374)
To me this is another attempt to show how expanding the scope of the system will affect it's failure rate.
It is known that the bigger the system, the higher the chance of failure to deliver it.
What is interesting is that this article brings another fresh look on scenarios where such rule happens:
replacing multiple legacy systems with new and supposingly better one.
Re:Overcomplexifying, Underdelivering
by ilaksh on Wednesday October 21, 2015 @04:01PM
(#10428228)
It makes sense to survey and try to think of ways to consolidate common functionality and data, but after a certain amount of that you need tackle modernization in small chunks, even though you will need a good way for systems to talk. And the business deal needs to be structured in small chunks too.
If you try to make a giant deal up front that is supposed to encompass all spending and tasks that many managers, engineers, or companies can think of, of course you are going to overrun. Or similarly trying to reproduce/refactor all of the efforts over several years of the predecessor teams of engineers and managers.
And another part of this is that people are bidding, which means you have to be in the ballpark of the other bids, and you probably need to be lower than the lowest bid, and everyone has an incentive to bid lower regardless of the actual scope because they have to make their bid attractive to have an opportunity to even actually examine the real requirements.
But its even harder because estimating even a lone software project is very difficult. Its hard to sell realistic budgets, since business people are often used to interacting with relatively simple to use software like Excel, where it might take a day to figure out the most complicated part of a spreadsheet for a certain purpose. By the way, don't underestimate the power and real value of spreadsheets to business people.
Re:Overcomplexifying, Underdelivering
by RankingMember on Wednesday October 21, 2015 @01:54PM
(#10427174)
This reminds me of the F-35 program as a whole. By trying to make it do everything, they end up with a plane that does nothing particularly well (and is prone to technical problems).
Re:Overcomplexifying, Underdelivering
by CaptSpify on Wednesday October 21, 2015 @12:44PM
(#10426639)
This is why I hate "all-in-one" systems. Systems that try to do it all, end up being hard to work with, and just introduce multiple points of failure. IMO, it's better to decouple systems and have each of them do one job, and do it well (https://en.wikipedia.org/wiki/Unix_philosophy). Then moving things around isn't as complicated or impactful.
Re:Overcomplexifying, Underdelivering
by rm445 on Wednesday October 21, 2015 @04:06PM
(#10428265)
That must be the case, right? Software being as diverse as it is, probably a creative individual could sit down and name 700 types of software application, but more likely they were numerous variations of CRUD, shuffling information in and out of databases, managing who could access what information, and presenting it in various ways.
Clearly there is something very wrong at the planning level of these big, failing, IT projects. Perhaps the temptation to begin by writing down all the commonalities between different software systems and seeing how one big system could do everything is part of the problem. But it's also easy to see how it is tempting to do so.
https://www.youtube.com/watch?v=rI8tNMsozo0
A lot of problems in software arise from choosing the easy route (i.e. gem install hairball), with no regards for the complexity it could bring into your project. Do this enough times and your project could easily end up like one of the projects described here: a massive mess of complexity that's excruciatingly difficult and risky to change and improve.
Some make serious money out of scope creep in IT projects. Sometimes I think that this must be how the major consultancies are sustaining themselves.
Someone wrote that the /average/ IT project ran 10x over budget. This would be completely unacceptable in any domain but IT.
All but #1 have a structural incentive to make projects bigger the same way a mechanic has a structural incentive to "find new problems." The client always has final say, but people underestimate how hard it is to keep saying "no" to multiple reputable firms all warning you against going too small.
That said, this kind of swindling is far from ubiquitous. There's a philosophy that blindly upping scope is shortsighted, and you do better in the long run by actually representing your client's best interest. Where I worked, I found it pretty easy to surround myself with people who thought this way.
But more realistic initial estimates won't win the contract for your company.
I don't know about the legal situation in the US, but in the EU, the lowest bidder gets the contract. So everybody has the best incentives of making a (sometimes even ridiculously low) initial estimate, and then find reasons to charge more.
And such reasons are always to be found, since requirements won't remain stable over the course of a multi-year projects. They never, ever do. And any change can be sold as an expensive scope creep that costs a few million USD.
If you try to make a giant deal up front that is supposed to encompass all spending and tasks that many managers, engineers, or companies can think of, of course you are going to overrun. Or similarly trying to reproduce/refactor all of the efforts over several years of the predecessor teams of engineers and managers.
And another part of this is that people are bidding, which means you have to be in the ballpark of the other bids, and you probably need to be lower than the lowest bid, and everyone has an incentive to bid lower regardless of the actual scope because they have to make their bid attractive to have an opportunity to even actually examine the real requirements.
But its even harder because estimating even a lone software project is very difficult. Its hard to sell realistic budgets, since business people are often used to interacting with relatively simple to use software like Excel, where it might take a day to figure out the most complicated part of a spreadsheet for a certain purpose. By the way, don't underestimate the power and real value of spreadsheets to business people.
The only way that could possibly work is if the original systems are grossly repetitive and unoriginal (variations on 2 themes, perhaps?).
Clearly there is something very wrong at the planning level of these big, failing, IT projects. Perhaps the temptation to begin by writing down all the commonalities between different software systems and seeing how one big system could do everything is part of the problem. But it's also easy to see how it is tempting to do so.