I myself have never seen Agile work well, not even close. I saw plenty of make-believe and head-in-the-sand and as mentioned in earlier comments, insane amounts of cargo culting.
I really like the way Dave Thomas put it: https://pragdave.me/blog/2014/03/04/time-to-kill-agile/
He also made a talk about it: https://www.youtube.com/watch?v=a-BOSpxYJ9M
These days this is the only methodology I follow and it works wonders, anything more I feel is just noise. At the end of the day nothing replaces actual engineering and understanding what you're doing.
My main gripe is most organizations I've worked in were usually in the transition between waterfall and some form of agile. The #1 issue was the people who sold the corporate execs on agile being faster and more efficient than waterfall have set unrealistic expectations as to deadlines and release dates.
Suddenly taking 12 months to build that app you were thinking of developing via waterfall, just got cut in half because everybody thought agile is so much faster, when in reality, it really isn't that much faster at all - just a different methodology, which a lot of people forget.
I have yet to work at a place that has released anything any faster than how it could be done using traditional waterfall methodology.
My last point about agile is that I've found it doesn't scale well. When I used agile in smaller teams (10-15 people, 3-5 devs) it was very effective. As you scale up (50+ people, 15+ devs) things slow down, inefficiencies are created because instead of several devs working on larger chunks, you have several dozen devs working on really small parts that get coded completely differently, then sometimes don't work together and it causes multiple iterations of design that wouldn't happen in waterfall. There is an advantage with us devs seeing the larger picture and coding to that, instead of this little component that gets coded 10 different ways.
In the XP teams I've been on, everybody rotates between stories and swap pairs daily, just to avoid the individual code ownership and fragmented coding styles you mention. Works really well.
> unrealistic expectations as to deadlines and release dates.
Also, one of the core agile ideas in my world is that there are no deadlines or release dates for specific features.
100% agreed. This is one of those things that so many companies can't even conceive of. But once you get people converted, it works so much better.
This means the "no deadlines" wet dream is usually a recipe for "oh shit, we ran out of money" for most projects. Also programmers are usually big babies; give them unlimited time and they'll give you unlimited bloat. Other than doing R&D it usually is impossible to have no deadlines.
Most people are not doing R&D, but when you really do your government usually has programs to finance it. That's precisely how you can afford to not have deadlines because your company is paid to advance the state of the art. And even then you usually get paid after the project is over and after you've filled a bunch of forms to clearly state the problem, your solution as well as existing work in the area and why it did not apply to your case.
Or if you're doing servicing work you get paid by shipping at predetermined milestones. Building software is a LOT more complex than just writing down the code and too many developers are clueless when it comes to politics, economics or management.
I used to think the same when I was still new to the software industry. It took me years to understand theres a whole world around programming required for programming to happen in the first place.
This is why one part of the Agile Manifesto that isn't talked about as much as it maybe should be is about how you need to build trust with your customer by constantly delivering value. Demanding that trust from strangers is hard sell, and it probably should be.
The big problem with deadlines is that it's impossible to predict how long something will take, so you end up in a game of lies and bluffs to create and take away buffers to account for the uncertainty, which leads to horrible relations between the dev team and it's customers. Also, it falls apart, when priorities change.
Deadlines are not that hard when you properly plan to reach them and give yourself a large buffer of time for debugging and QA. You just need to properly analyse the requirements, the existing solutions in your problem's domain, what is known and unknown (keeping meta-ignorance in mind - you dont know what you dont know) and then re-analysing it all every time the requirements change, even slightly.
This means you're going to spend a lot of time upfront before coding starts, and the first deliverable will be months away. Solid foundations yield by far the most agility to any project I've ever seen, and continuous delivery prevents these foundations from being built in the first place. You definitely won't see any proper abstractions at such a pace, what you're most likely to see is an endless sea of indirections.
Same for testing, I've seen many teams starting to write tests for requirements that weren't well understood yet. Then they start piling code up until the tests passes, and finally wonder why there's tons of new bugs in Jira every week. All the tests passed! The type checker passed! But now there's no time left for debugging and the project gets over time and over budget.
The smoothest servicing projects I've ever done were with customers understanding the nature of our work and not minding a long time window before the first delivery, not interfering with the work. The worst servicing projects were having two weeks sprints where the managers would bend over to every single customer demand for fear of losing the contract at the cost of burning out the team and having no quality whatsoever in the finished product.
You get trust with new customers based on your own track record of successfully shipped projects, not continuous delivery over arbitrary milestones. And then you can manage the customer, instead of being managed by them. They pay you because they can't do the work in the first place after all.
Note that we are discussing "no deadlines or release dates for specific features", not just a magical world without consequences.
If you are looking for how this works, Poppendieck's books on Lean Software Development go into both the theory and practice. But the basic notion is that you control costs via scope control, waste reduction, minimizing WIP, and continuous improvement. One still delivers things quickly and efficiently; it's just that deadlines are not used as a control mechanism.
Once you're through Poppendieck's work, I recommend "Priniciples of Product Development Flow": https://www.amazon.com/dp/B007TKU0O0/
Honestly, I think you eventually can get substantial speed gains. But you mostly get them by cutting out waste. Doing that takes a fair bit of time and work. And not just work, but commitment to personal and cultural change. People don't do that under strong time pressure; instead they are inclined to revert to previous behavior.
Besides, what's to say there wont be overlap in these newly separated services that will generate work in double because teams dont talk to each other? Or maybe you'll generate two average solutions instead of a better unified one.
Even with multiple teams working on different aspects of a project, you still need yet another team to maintain the global vision of the thing. To make sure everything is consistent, to avoid work duplication, to consolidate solutions and abstractions and whatnot.
Its fair to say the larger a development team is, the harder it will be to manage. Its even more true with large teams of average developers, which really is the norm when you think of it.
At the end of the day, communication is key.
Bingo. Usually when you ask them what agile processes they are following they just mumble something about daily stand ups (which I'd argue are not agile at all).
I've seen agile work. But it's more about automating builds, having continuous (or near continuous) deployments, having database migrations to easily integrate others changes, etc.
My evaluation of how agile a company consists of "When a trivial bug is reported, how long until the fix is in production?". If the answer is greater than a day then they aren't agile.
A better question to ask an "Agile" team would be: "what problems are you trying to solve?" - most of the time they haven't taken the time to properly analyze the problem in the first place, even when the project is near completion. They just run with dumb user-stories and start piling up Jira tasks with no clear vision.
At this point I don't believe Agile can work whatsoever. A success rate of one in a thousand projects, which usually happens to be a dead-simple CRUD app anyways, means absolutely nothing, other than it has failed in a spectacular way. If software development was an actual science instead of a pop culture, the scientific method would've thrown Agile out the window decades ago.
Your last two sentences almost exclusively applies to web services, which is a tiny fraction of what software engineering can do.
For me real agility is how much effort you're willing to spend upfront to properly analyze problems before attacking them, how quickly do you adapt your solution when new facts become known.
Its dead-easy to have agility at the beginning of a project, what matters is still having it 2 years down the road and more. Continuous deployments, database migrations, automated builds and whatnot do absolutely nothing to help you reason about the simplicity of your codebase - which is what yields true agility.
> Continuous deployments, database migrations, automated builds and whatnot do absolutely nothing to help you reason about the simplicity of your codebase - which is what yields true agility.
The help in the sense that you can make and deploy changes seamlessly. For understanding code the unit tests that go along with them also help tremendously, if they are well written unit tests anyway.
Besides, you set them up once and then forget about them, happy with your day to day use. But actually fighting complexity and making sure you're still solving the problem at hand and not bloating things up requires near constant critical thinking, something no tools or processes will ever be a replacement for.
That's what I meant :)
I pretty strongly disagree with this. Simplicity in agile projects is achieve through continuous improvement of the code. All of those things make refactoring cheaper and easier, which means you can keep your code base in good shape as domain and business needs change.
Without those supporting practices, people get scared to change anything. And once they get scared, bit rot sets in.
This is also called "emergent architecture" and its a big lie. You'll at best reach a fraction of the quality you'd have by thinking about said architecture upfront with twice the development time - it works for CRUD projects simply because such projects are dead-simple to begin with.
How are these practices helping us reason about the simplicity of the codebase? How are the people dependent on your code going to handle all of your frequent refactors? What does continuous deployment even mean outside of the web? How do you refactor something as big as a game engine with 10+ projects built on top of it?
More importantly, how are you going to add proper abstractions to an existing project? No refactoring tool will ever do that, it cant be automated and the more advanced a project becomes the harder they become to introduce in the codebase. I've seen plenty of "best practices" projects that were just huge hairy balls of indirections. Yet everyone on the team felt good about it because none knew any better.
You can't judge practices based on how they work for one particular area of software development. Especially when said area is one of the simplest out there.
You know exactly nothing about what I've based my opinion on. Because all you've done is jump in and rant about your beliefs.
> thinking about said architecture upfront with twice the development time
Design quality is limited by the information you have, and no amount of thinking creates information. The beginning of the project is when you have the least information about everything: domain, users, user needs, business needs, technology, team. Everything.
You mostly get information by trying things in the world. And actually trying things in the world is the only real test of ideas that come from thinking.
If your methodology requires understanding everything up front, then it's basically useless in the real world. Even if you could perfectly understand your intended market and use case, that will change over time. But you can't, because your competitors are always plotting to beat you, and they will not share their plans with you.
Amazon, for example, has been in operation 20 years, and could easily be going for decades more. If in the early days they had tried to write code capable of dealing with their current needs, let alone their needs from 2050, they would have been idiots. Pure up-front design is an unworkable fantasy.
> More importantly, how are you going to add proper abstractions to an existing project? No refactoring tool will ever do that [...]
Refactoring existed before automated tools, and mostly exists beyond it. Tools help, but it's mostly a human activity. Try reading Fowler's book on this, and his subsequent writing. You can't effectively criticize something you don't even understand.
For example, I think a focus on quality is necessary to enable real agility, so I'd look for unit tests, acceptance tests, continuous integration, continuous deployment, pair programming (or fast-cycle code reviews).
And for looking at results, I look for things that are hard to fake, like release cadence, amount of work in process, and unit of work size. So I'll ask, "When was your last release? And how about the release before that?" And I'll want the difference to be small. (My favorite place to work released multiple times a day.) I'll ask other questions like, "What are you working on? Who's working on that? When did that start?"
Another thing I'll measure teams by is the extent to which they're focused on problems vs features. Good teams ship things that solve problems (and make sure that they actually do). Bad teams ship features because that's what the spec says or what the boss wants.
I was hoping for a way to look at a job posting without going in and ask people, but I guess that's not really that feasible.
Maybe I just need to devote a few weeks to chat with a lot of companies when I'm ready to go back to work.
Good luck with your search! If you're in the Bay Area, feel free to drop me a line when you go back and I can recommend some places.
Agile helped. Many programmers appear to find unions distasteful, so they needed some other way to try to protect their jobs from outsourcing. Agile claims "face-to-face" is most efficient.
But if the team is a "black box" with their only contact with the rest of the firm being mediated by the Product Owner, then that team could be anywhere. I always saw Agile as a sly trick to make outsourcing easier by packaging the team into a self-contained unit.
Also, that black box team is by definition not "Agile" because the engineers are not face-to-face with the users, nor communicating with the users daily during development.
And if your "daily" communication with "users" involves specifications and talking to managers, it doesn't matter whether you're in-house or outsourced.
It looks different for modern consumer products, because your software is typically a small part of someone's life. They don't want to talk to you daily. Instead you have a product manager acting as proxy for the users. But if you have good instrumentation, you're still getting user feedback daily.
Even in the consumer context, you can talk to real users frequently. At my last startup, we had user testing every Tuesday afternoon. We'd book 4-5 sample users every week, and we'd always have some question we wanted to answer. And plenty of places have small beta tester groups that provide fast feedback on new features.
That all depends on a fast release cadence and a tight relationship between product management and developers. But once you have it, it's addictive for product managers. Oursourcing basically becomes impossible, because their turnaround time for answering a question drops from seconds to hours, and the turnaround time for shipping a small change goes from hours to days. This forces batch size back up, which is not only vastly less fun, but also reduces ability to explore the product space and find high-value business wins.
Every time I've been asked about outsourcing, I've asked, "Will you send your product managers there too?" 99% of the time the answer is no, because they can see how that would impair the information flow. The trick is getting people to extend that desire for connection and fast iteration past the managerial tier.
Everybody meant well, and everybody was very smart. In retrospect, though, I think the big mistake we made was thinking that if our hazy principles (as expressed in the manifesto) collided with typical American managerialist business culture, our principles would win out.
In practice, we ended up with what I think of as "mini-Waterfall with Agile labels". Release cadence is faster than with traditional Waterfall. But it is still generally a top-down, push-oriented, document-heavy system. Line workers still seem just as miserable and disempowered as before.
I personally have used Kent Beck and Mary Poppendieck's work to create some extraordinary teams and some great working environments. So I think the principles and the original insights have deep merit. But I recognize little of that in the typical "Agile" environment.