Can I ask why do you deliver stories before refactoring, if that's what you're saying? We'd tend to do necessary refactors as part of features or bugs when we touch related code and rarely have pure refactor chores.
Also, why do you have to _sell_ code quality? At least where I work there is a pretty strong divide between what to build and how to build it. There is two-way trust between the product manager and engineers that they know their domain and when they say "this is important right now", it's probably the right thing. After that it's just a matter of communication and prioritisation between the the two sides.
I could easily see how this might fall apart if either side was not trusted or not doing a great job, but I guess it would probably fall apart no matter what development process you followed if that were true.
But the organisation is slowly carving our monolith into microservices (Because buzzwords will save the day! <sigh>). I do feel microservices lend themselves better to agile, as the programmatical boundaries a good dev should naturally code within become rigidly defined. And people can effectively do their own thing within their "box".
IMHO it shifts the problems to the layer above, papering over the cracks. (Badly written code performing badly? Whack it behind a gateway and spin up some more containers!)
Even with microservices you're gonna have to define that API at some point (and now we can have as many as we like behind the scenes!), why not at the start with a holistic view... like, a design!
"As a user I can see a login page" - ok here's a static html page with two input boxes and a submit button. "As a user I can log in" - implement that slice of the backend, all in one hit with no eye on what other things might need from it (Dont get me started on YAGNI... YMNI, imho and you'll miss it when it isn't there! Especially when there's "no time" because it's not a feature with perceived customer benefit which take priority) And it goes from there. What I would prefer is more like "before we can even think about showing a login page to a user we should have..." which is more bottom up, and doesn't look good to a customer (or management).
This should be a hard sell. The code in production works. Why does it in need a rewrite? The fact it needed a rewrite in the first place doesn't really reflect on scrum but on poor engineering and management practices.
This might be what I have the biggest problem with. To a customer, it works. The customer is happy, they pay their money, which makes management happy, and we all get paid. So the site takes 10 seconds to load on certain pages, who cares! It works, right?
Behind the scenes, there's a fleet of web servers burning CPU so add some more! A database that's overwhelmed, throw some more CPU/mem and faster disks at it! Maybe optimise some queries if you can get it into sprint. But a fundamental problem with the implementation itself cant be fixed, because "it works". Want to leverage some slave dbs for reads? Sorry, no time for that because unpicking the backend would take too long!
It's backwards. In my idealised view: Good design first = better code = less resource hungry = less cost for servers = support more users = more revenue = less time spent working around crap = more time to implement features! = more revenue = better wages. Everyone wins.
Imho, agile doesn't help make this happen.
If you really have a fleet of web servers burning CPU and you arne't tracking the costs, then I guess "it works" until someone figures out there isn't any money left in the coffer.
There is a lot of value in working software, and unless your engineering infrastructure expenses seriously outweigh your engineering payroll costs (which does happen, but is hardly commonplace), CPU efficiency isn't the be-all to end-all and focusing on it may well prove to be counter productive. If no one actually cares about your page load times, it may very well be that it doesn't matter. You can spend a near infinite amount of time & resources trying to get optimal performance (talk to someone who does high-frequency trading to get a sense) well past the point where the costs pay off.
There is tremendous value in "it works" that engineers generally struggle to fully factor into their thinking (I come by this wisdom honestly). People have written about this extensively (e.g. https://www.joelonsoftware.com/2000/04/06/things-you-should-...). That said... the notion that you can't fix actually serious and counter-productive problems with the product because "it works" sounds not only counter to Scrum, but really counter to any software development methodology I've ever heard of.
What if the costs are fixed? (own servers, unmetered bandwidth) The problem shifts down the line when you want to add a new product and the first product is consuming all your resources. Refactoring then becomes a necessity, but there's no capacity for it because the company is focused on shiny product #2, while also adding shiny features to product #1. Now fire a few people to save money, leaving the remainder short handed, who get stressed and demoralised. But that's ok, because "it works!". Why cant we design first, lay down a good foundation, then add features. The business makes more money, people keep their jobs, instead of spit-balling "user stories" at an application which becomes a giant spit-ball.
> CPU efficiency isn't the be-all to end-all and focusing on it may well prove to be counter productive > You can spend a near infinite amount of time & resources trying to get optimal performance
I used CPU as a broad example. What happened in my case was there was no performance. At release it was unusable. What stuck in my side was I knew it was going to happen, and had been calling for refactors. But if you cant get them prioritised because you cant justify the customer benefit (it's not released yet, so nothing has fallen over yet) vs features! features! features! it then becomes a bodge job /after/ release. eg. Instead of "refactor this code to make it do fewer queries" it's "lets shove the results of these queries into a memory cache." This satisfies agile, you've iterated and made things "better" - and it fit into a 2 week sprint! - but it feels backwards.
I was going to go on and talk about quantity vs quality, but a quick fact-check google revealed the evangelists with their "Agile parables" (it even sounds cultish), which are in no way perpetuating Agile to sell support tools, say I'm wrong.
I need to thank you for making me realise it's time for a career change. After all this time Agile has failed to convince me, I'm inclined to conclude it never will. But should the next methodology of the moment come along that emphasises quality over quantity, I reserve the right to shout "I told you so!"
Again, your problem isn't coming from the methodology. The priority on quality comes from the context, not the methodology. I have a friend who does presentations on agile all the time and his team prioritizes quality pretty much over anything simply because it allows them to develop faster.
Quality got worse.
In this scenario it's the fault of agile, as the reason for doing agile was to increase quality.
It's a tool/methodology that can help you increase quality.
If quality isn't a priority, a methodology that helps to better align your work with your priorities isn't going to improve quality.
Feeding that back through the process is pointless when it inflates the complexity of future stories, because they are less likely to be prioritised, and the rot festers.
Peer review is a good way to improve the quality of peers, either though the natural education/knowledge sharing, or simply the transparency that helps you remove low quality peers and find better ones. It's not a cure all, but it's hard to imagine a successful team that wouldn't benefit form it.
Some turn up in the morning, however, work on their task for the day and go home. If they reimplement a chunk of the codebase somewhere else, because they feel they need to be writing more code to complete their task, fine, the story still passes. "I didn't know that was there" is not an excuse, it was probably covered in planning but they forgot. It's not going to get refactored, because it works! Peer review in this case goes one of two ways:
1. Bad quality peer sees criteria are satisfied and approves 2. Good quality peer sees all the duplication, some other bad practice thrown in for good measure and disapproves
In 2. say the story has already carried, pushing more work back, and hard deadlines are looming. They will be forced to begrudgingly approve the work, knowing they've just allowed a good chunk of cruft to enter the codebase. They will try and educate (assuming they've not lost the will to live - this might be the 100th time for eg), to prevent it happening again. But it will still happen, because bad developer is bad.
Forgetting the looming hard deadline: If the story carries over so what? We'll retro and add some bloat to the process to try and avoid it happening again.
What isn't explicitly said is "this dev is committing crap and the competent people are getting demoralised", it's the process that gets "refined" not the people. Agile is papering over the cracks and nurturing the codemonkey's behaviour.
But none of this matters, as the deliverable has been delivered. Now on to the next story!
Thankfully I dont do agile any more, but I'm close to several teams that do. My fondest memories are:
No over-arching design (Implement feature after feature however the hell anyone likes) Once said features are implemented, taking the time to "refactor" (read: completely rewrite because the code was so bad) becomes a really hard sell.
Thus, new features get progressively harder to implement, as you touch a deeper cross-section of the code to try and fix stuff semi-covertly, without a formal refactor.
"Tech Debt" was done in sprint scheduled after phase of dev sprints. After, say, a year of development, lets fix all the problems in 3 weeks... When you're dealing with procedural php masquerading as OO, MVC with business logic everywhere it shouldnt be, and abuse of inheritance such that you cant throw an error in a command line because you're not a logged in browser user, who's had their language queried from a database, and doesnt fall back to default when the db isnt there... 3 weeks isnt gonna cut it!
Introducing peer review was supposed to sort that, but bad or inexperienced devs didnt know they were doing shoddy work (it met the criteria after all..), and good devs became impediments because so many stories needed redoing. Demoralising.
We were probably "doing agile wrong". But it's like a religion: open to interpretation. In my experience it's a tool used to extract maximum customer benefit at the expense of the developer's soul, under the pretence of "self managing team" - ie. "You brought it on yourself".