News for hackers. Stuff that matters.
Re:Ask HN: Have you ever worked on a product that was killed... by PaulRobinson on Tuesday January 17, 2017 @12:37PM (#13419382)
I was CTO of a company that had a two-week outage due to technical debt. I didn't sleep much for any of it. We fixed it, and we'd lost about 30% of our subscriber base in that period. The company took on new funding to survive, invested that in a new set of products, and shuttered the old stuff just to stay afloat.

I am currently working in a business where there is a nearly 8-year old Rails app (600+ models, 250+ controllers, 400+ libraries, LOC around 60k), that sits at the heart of everything we do.

The company is struggling to grow and believes the cause is that engineering is slow. We have asked to refactor this code base multiple times, and point to the technical debt as the cause features that should take a day to implement taking between 3-4 weeks, typically.

It is only recently that the penny has finally dropped and they've realised if they don't invest in replacing this thing (there is too much technical debt to fix, we're calling bankruptcy and moving to a brand new architecture piecemeal), the business is likely to fail within 1-2 years.

That means my current employer is likely to go bust because of technical debt within 2 years max unless we become really good at fixing this.

We are optimistic.

We have to be, right?

Re:Ask HN: Have you ever worked on a product that was killed... by sydd on Tuesday January 17, 2017 @02:25PM (#13420317)
IMO this is the price to pay for a dynamically typed language. 60K LOC is not much in a static language, you can use tools to refactor it easily or to visualize the control flow. But with a dynamically typed language? Its a nightmare. You change one thing and cannot possibly know what else could have gone wrong.
Re:Ask HN: Have you ever worked on a product that was killed... by PaulRobinson on Wednesday January 18, 2017 @06:08AM (#13425530)
The legacy app has > 80% test coverage. Refactoring is still slow because there are all sorts of business assumptions put into place that add functionality without ever questioning the need for it.

Basically, for a long time, the company never really re-evaluated what it had learned and spent time trimming things down, so as a result there is this ungodly mess. At the heart of what the business does, there is no real need for more than a dozen models. So why do we have so many more? Nobody ever refactored away stuff we didn't need any more, and so weird things happen.

There is also a coupling issue that is endemic to all monoliths. We're moving to a micro-service architecture with clean domain separation, and we'll probably go to 1/10th of the code base in LOC terms within 12 months, even if we move some of that functionality into Go, Java or Python services (all options).

Re:Ask HN: Have you ever worked on a product that was killed... by sydd on Wednesday January 18, 2017 @07:39PM (#13431922)
Unit tests just make the mess a bit easier to solve, but are far from a perfect solution. Wanna move and rename a function? Do it, then spend hours fixing your 10 broken tests, writing new ones, and test you app for hours because unit test dont cover integration. A static language + an IDE does it automatically within seconds.
Re:Ask HN: Have you ever worked on a product that was killed... by pizza234 on Tuesday January 17, 2017 @03:30PM (#13420879)
It depends.

I work on such type of codebase, but we have a fully covering testing suite, so applying changes is not a problem (interestingly, I've just realized that the line count of the testing code is 50%+ more than the base application code itself).

So ultimately I think company culture (that is, emphasis on automated testing, for dynamically typed languages) is the crucial factor.

Re:Ask HN: Have you ever worked on a product that was killed... by 72deluxe on Tuesday January 17, 2017 @12:55PM (#13419550)
I would say that 8 years to write 60k LOC is slow. I worked as the sole GUI software engineer for a hardware firm and wrote > 100K LOC in 3 years, not including the test projects proceeding the actual real project. This was in C++, and included client/server stuff, entirely custom resizeable GUI, OpenGL 3D graphics and modelling of 3D assets and textures etc too. And getting it running under OSX + Win32, fixing issues on both.

And that wasn't a stressful place to work with insane deadlines - it was fairly relaxed for the most part.

Re:Ask HN: Have you ever worked on a product that was killed... by gizmo on Tuesday January 17, 2017 @01:03PM (#13419624)
You can't compare C++ with Ruby code. Rails code especially can very easily become a hairball where everything happens "somewhere else". You don't have static type checking or other compile time hints to figure out what is going on. You can't see which functions are called from which call sites. Refactoring tools? Forget it. Ruby is a very compact and flexible language, but if you're not disciplined you'll pay the price for it.

A 60kloc C++ project is small and easily manageable, a 60kloc Ruby hairball can drive a person insane.

Re:Ask HN: Have you ever worked on a product that was killed... by RangerScience on Tuesday January 17, 2017 @01:05PM (#13419649)
Rails code. You get a lot of bang for your buck. That, and the "greater" metric for the project complexity is the stupidly high number of controllers and models.
Re:Ask HN: Have you ever worked on a product that was killed... by golergka on Tuesday January 17, 2017 @01:07PM (#13419661)
LOC differ not only by the language or technical difficulty of the task, but by business requirement difficulty as well.

If, for example, they're in banking and finance, and those LOC deal with fine details of tax code... Oh boy.

Re:Ask HN: Have you ever worked on a product that was killed... by wheaties on Tuesday January 17, 2017 @02:40PM (#13420446)
As a former C++ dev, you can't compare Ruby LOC to C++ LOC. Just 4x the Ruby and then it becomes more fair. C++ is just verbose. #import <algorithm.h> doesn't fix everything blocks fix.
Re:Ask HN: Have you ever worked on a product that was killed... by PaulRobinson on Wednesday January 18, 2017 @06:09AM (#13425535)
This is why LOC is a terrible metric. I was using it as a barometer as most ruby devs would see 60k LOC and go "Ooooookaaaay...."

If anything, we've gone too fast and not spent enough time going back and understanding what we really need to keep.

Re:Ask HN: Have you ever worked on a product that was killed... by luhn on Tuesday January 17, 2017 @12:49PM (#13419503)
Can you expand on how technical debt caused a two-week outage?
Re:Ask HN: Have you ever worked on a product that was killed... by TheCoreh on Tuesday January 17, 2017 @12:54PM (#13419541)
If I had to guess, I'd say it's probably an issue with the build/deploy system. Perhaps someone deployed a broken build, then tried to revert/rollback, and realized that the previous version didn't build "cleanly" anymore.

This could happen if you have a lot of dependencies, switched compiler versions but left the binaries "in place" and deployed changes incrementally.

Re:Ask HN: Have you ever worked on a product that was killed... by RangerScience on Tuesday January 17, 2017 @01:06PM (#13419656)
Yet another reason I swear by Heroku - you can rollback to the actual prior release, not rollback the code and try to rebuild.
Re:Ask HN: Have you ever worked on a product that was killed... by ardivekar on Tuesday January 17, 2017 @01:28PM (#13419818)
Doesn't AWS do this too?
Re:Ask HN: Have you ever worked on a product that was killed... by RangerScience on Tuesday January 17, 2017 @01:50PM (#13420001)
Which service? EC2 doesn't - at least not inherently, there might be a way to snapshot the machine's state; I'm not sure.
Re:Ask HN: Have you ever worked on a product that was killed... by luhn on Tuesday January 17, 2017 @01:36PM (#13419885)
Depends on which of their 150 services you're talking about. You can't do it on CodeDeploy.
Re:Ask HN: Have you ever worked on a product that was killed... by PaulRobinson on Wednesday January 18, 2017 @06:04AM (#13425508)
There are two problems with giving a full answer: firstly I'm still under NDA for some of that, and too much detail would breach that; secondly, my exact memory of it is limited.

In short, my predecessor had attempted a move to SOA without understanding dependencies, circuit breaking and failure modes. This would then cause scenarios where the entire front-end would fail to render on a single down-stream service taking a little longer than necessary.

When identifying how to stop that happening, I discovered a large number of comments tagged "TODO" with statements like "Refactor this when we have time" or "We need to find a way to do this better".

Further down on the downstream services there were rather esoteric SQL queries doing large joins that nobody had done a query plan on. It was hard to identify these because the ORM had been trusted to do magic, and it was happy to do so, but there was a point where it was not apparent _why_ these joins were happening, but when you found the code, there were more comments "This needs improving", "We should refactor this", etc.

We were able to get something back quite quickly with liberal application of indexes, and it took us a day or two to refactor the queries enough to mean response times came down, but the error rate was still > 20%, and it was random, so 1-in-5 page loads of the front end service would fail.

We refactored the code to circuit break and handle degraded services better, but that took a few days, and then we started working down to the back end service and figuring out the final steps.

It was a small team looking after legacy code that everybody knew was a bit messy.

A few weeks before this code was shuttered, I heard from a friend that some of our content did not render at all on certain Android devices. I identified the cause as a half-finished refactor (again, my predecessor), that had never been finished because he had been pushed to work on something else. This caused a dramatic decline within a key market segment that resulted in declining ad revenue, subscriptions and overall viability of the business.

Basically, when you start something, finish it. If you find yourself putting in comments like "We should refactor this" anywhere in your code base, and you're doing so because the business is pushing you to work on new features, you have a massive problem culturally that is going to cause a rise in technical debt that raises risk to revenue.

All technical debt ultimately will lead to problems that the business will see on balance sheets, but they will rarely successfully identify the cause as being technical debt because they can't see, understand or rationalise it. They think it's engineers being grumpy idealists.

People play too fast and loose with the concept of "MVP" for my tastes, and it's a problem I see over and over again. The risk of that is, long-term, it will cause business failure.

Re:Ask HN: Have you ever worked on a product that was killed... by nicostouch on Wednesday January 18, 2017 @06:41AM (#13425670)
Solid lesson. Thank you for sharing.