News for hackers. Stuff that matters.
Re:TSB Bank: Botched upgrade has left customers unable to ac... by dx034 on Tuesday April 24, 2018 @07:08AM (#16911534)
Why didn't they roll back the update as soon as that was discovered? Would've caused another downtime of a few hours but better than the current chaos.
Re:TSB Bank: Botched upgrade has left customers unable to ac... by isp on Tuesday April 24, 2018 @07:51AM (#16911753)
I have no inside information for TSB, but I have worked on other large migrations before at $DAYJOB.

Based on the original planned 50-hour outage, I suspect that TSB chose a "big bang" migration: all-or-nothing, no rollback possible. These are technically easier (so cheaper) to develop, but far far riskier than taking a phased approach.

I have successfully argued against such approaches in the past, due to the high risk of catastrophic failure if it goes wrong (i.e., exactly what has happened to TSB).

I regard TSB's failure as an in-the-making textbook example of "how not to do it".

Re:TSB Bank: Botched upgrade has left customers unable to ac... by mcroft on Tuesday April 24, 2018 @08:35AM (#16911979)
Often, these kinds of migrations are the only option due to previous cost cutting/money saving decisions - likely due to pressure from the business and nothing to do with any technical reason.

I remember one migration which involved lots of internal services which were all tightly coupled, meaning updates across the whole backend, not just the part that needed it and no way to do a phased rollout. It was made worse by the fact that the update was to the platform and that depended on a DB upgrade, but that upgrade was incompatible with the old version which meant everything had to be done at once.

They had no disaster recovery plan. They got it done, but I'm sure a lot of people involved went grey early thanks to that nightmare.

Of course the business had no idea of the utter mess that caused all of this and continued to make harmful decisions in the name of saving money and further underfunding the IT department.

Re:TSB Bank: Botched upgrade has left customers unable to ac... by ams6110 on Tuesday April 24, 2018 @10:20AM (#16912759)
Even with tightly coupled systems, ultimately it's all bits on the wire. You can always develop shims or intermediaries to perform a phased migration with rollback options. Of course that takes time and costs money, which is why some people take the damn the torpedoes approach.
Re:TSB Bank: Botched upgrade has left customers unable to ac... by isp on Tuesday April 24, 2018 @09:53AM (#16912549)
Fair general point. (Though it is often - if not always - possible to do some sort of dual-running, to mitigate risk. Albeit at much greater expense.)

In the specific case of TSB, I have my suspicions about the reasons for cutting corners in this migration, given: https://www.thetimes.co.uk/article/missed-deadlines-leave-1m...

Re:TSB Bank: Botched upgrade has left customers unable to ac... by dhimes on Tuesday April 24, 2018 @10:19AM (#16912749)
It's bad enough to not be able to roll it back, but to also not thoroughly test? Unfathomable.
Re:TSB Bank: Botched upgrade has left customers unable to ac... by dx034 on Thursday April 26, 2018 @10:51AM (#16932363)
The problem with core banking systems is that they're nearly important to test. There's no distributed system, you always need one single source of truth for the whole bank (the ledger). Moving part of that is only possible if you keep shadow accounts in the old system. That's also why live migrations aren't possible and need at least one night downtime.
Re:TSB Bank: Botched upgrade has left customers unable to ac... by dhimes on Friday April 27, 2018 @07:50AM (#16940015)
Can't you set up fake accounts with fake data? Random numbers? I do that all the time is a different context. I have a bunch of scripts I run that populate a database with nonsense. So not shadow accounts, but fake accounts.
Re:TSB Bank: Botched upgrade has left customers unable to ac... by tonyedgecombe on Tuesday April 24, 2018 @07:11AM (#16911543)
I'm not sure this is an option, this is all about the separation from Lloyds bank.
Re:TSB Bank: Botched upgrade has left customers unable to ac... by dhimes on Tuesday April 24, 2018 @10:17AM (#16912736)
This is exactly what I was asking my bank, TD Bank, after their recent upgrade was also a disaster. It looked like a fairly straighforward error from the client side- the login script was looking in the wrong place for my email and couldn't find it (they had my email, because they were still emailing me).

Tech support switched me over to a part of the site that had not been updated, and that worked for a while. Then it didn't. Back to the bank I go, with a new customer service representative, who really didn't grok the issues (new to the company?), and who became quite snippy.

Very aggravating.

Re:TSB Bank: Botched upgrade has left customers unable to ac... by codeulike on Thursday April 26, 2018 @07:24AM (#16930877)
Not so much an upgrade, as moving away from Lloyds system (they were previously merged with Lloyds as Lloyds TSB) and jumping onto a completely new system built for them.