These things were put together in haste in the 90's and Oracle keeps pushing them and their corporate customers keep these things going long past when the interface becomes utterly unfamiliar.

The blame has to be shared, however, with the battle-axe personalities of the people that use this stuff. They would like nothing more than to retire while still using the _exact_ same applications and dated interfaces they've used for 20+ years. Literally, these are default "grey-theme" java applications, with all kinds of inconsistent UI quirks and absolutely no shits given for discoverability or even being able to create hyperlinks so that people can share stuff without screenshots.

It's fine if it's the same people keep using these apps forever, but the problems start when new people get on-boarded. If it's like most places, the newbies will be given no training and the only help they will get would be from some dour in-house oracle-application jockey whose been doing the same thing for decades. That's when people make very, very expensive mistakes. We hear about this one because the sum was vast, but people make $10,000 mistakes on shit software from Oracle all the time-- that's partly why Oracle has a very deep bench of retained lawyers, I suppose.

I have to say Citibank deserves it.

Actually the history of this app is a little more complicated. In the early 1990s, Citibank decided to establish an outsourced software development group in India to rewrite their banking software from scratch–they established it as a subsidiary company called iFlex. From 2005 onwards, Citibank progressively sold a majority of its stake in iFlex to Oracle, who then renamed the company Oracle Financial Services Software (OFSS). OFSS is listed on the Mumbai stock exchange, but Oracle owns the majority of its stock.

So, rather than being a product which Oracle developed themselves and sold to Citibank, this is actually in some ways the opposite – a product Citibank developed themselves and sold to Oracle.

I used to work for Oracle. I never worked on OFSS software directly, but I did work on an implementation project for some of it. I never saw the internal banking UI, all I ever saw was backend stuff – LDAP, BPML, JVMs, databases, load balancers, firewalls, etc. Indeed this article is the first time I've seen that particular UI in my life. But I can tell from the visual appearance that it is not one of Oracle's more recent UI development frameworks. (I don't know if this is because Citibank is running an older version, or if there are parts of the product still running on a legacy UI framework.)

https://damieng.com/blog/tag/delphi/

When the 3D look became the norm for Windows apps, introduced by Office and, later, by Windows 3.1 (or 3.11), I used the DLL to 3D-fy my Windows app dialogs automagically. I also took some liberties with using Office graphic elements reasoning that a Windows or Office user would know the 3.5" floppy saves things.

Citibank developing in-house software then selling it to Oracle so that they can charge recurring licensing and support fees for eternity is incredibly short-sighted if you're Citibank or Manna from heaven if you're Oracle.

My guess though is the Citibank executives who signed off on this got promoted up and out of the department that has to budget for these licensing fees and support costs, and on both sides they received tidy bonuses for "increasing revenues".

Usually it's a win win, unless you value keeping control (which can be quite valuable)

My experience is that it's usually overvalued. An external, focused company will usually do a better job than an internal project. It's emotionally hard to give up control, but successful companies tend to focus on a few things, do them really well (areas of competitive advantage), and outsource everything else.

A best-of-breed external solution almost always beats an in-house solution.

I'd just never do something like this with Oracle. Their business model seems to consist of locking companies in, and then milking them.

Oracle paid Citibank over US$500 million for this company. There is no way a transaction that big isn't being approved by the CEO and board of directors of both firms. It wouldn't be under the control of the software licensing department. It would be controlled by the business development group in charge of M&As and divestments.

> Citibank developing in-house software then selling it to Oracle so that they can charge recurring licensing and support fees for eternity is incredibly short-sighted if you're Citibank or Manna from heaven if you're Oracle.

I have no internal info on the details of the Citibank-Oracle relationship (my role at Oracle was technical, not the kind of role where I would know that kind of business relationship stuff, and I never worked on the Citibank account either.) But, as @dagmx has already pointed out, I think it is highly likely due to the history of this software that Citibank has some kind of special licensing deal with Oracle which protects them against that sort of thing.

For Citibank that was US$500 million then, which they could either invest directly or return to investors. The opportunity cost of that cash has to be factored in.

It's also an a large ongoing liability to have a sizable in-house development effort when it's not your main business. Departments need to be staffed and run, business plans made etc.

For all of Oracle's flaws they're presumably going to have an incentive to improve the software, and more capital with which to do so from clients other than Citibank. For obvious reasons other banks would be more inclined to buy from Oracle than a direct competitor.

There's also often tax reasons for why it's preferable to buy a service v.s. maintain in-house software, and naïve back of the envelope math usually doesn't account for that.

This is the same ideology that led to companies outsourcing everything from facility management and cleaning services as the first victims to stuff as critical as IT operations.

Yes, it is a liability and likely also a higher cost (e.g. due to collective wage agreements) to do that stuff yourself. On the other side, you have the large liability of having no direct control over staff and work quality any more - you're entirely at the mercy of your contractors (who are incentivized to find not the best people, but the people willing to be paid the least).

This is important. Just a few days ago here on HN, there was a good discussion on how deliverables quality on seemingly "mundane" device chargers (in this example, one that was arguably for early Kindles) can vary wildly, while maintaining the same price point, with the sub-standard quality yielding more profits to the seller, and higher-quality units yielding less profits.

This is a impedance mismatch between what the buyer knows and values and what the seller knows and values. With a time slippage between the time the buyer makes a decision and when the buyer's organization figuring out the real ramifications. By the time an organization has outsourced enough to lose competency to not even know the impedance mismatch and time slippage exists or scope it if they're aware of its existence, sometimes they will find might be cheaper to in-source in the first place. My personal rule of thumb going in to evaluate these decisions is if the procedural complexity surrounding such software artifacts rises above a certain level, it is likely better to keep it in-house. Where that level is lays the art of business, it is largely experience-based.

Maybe they saved the US$500 million they were paid for this software, and they can use it to cover the US$500 million it just cost them.

However the OF technology is still supported by Oracle and while they themselves are trying to kill it, nobody is willing to migrate their core systems to something new.

It’s actually easy to sell shit software. The average person knows a lot more about cars, for example, thank they do software. I’m sure Oracle makes huge money on name alone. The IT manager doesn’t want to take a chance with some lesser name. Just get Oracle, then nobody can blame them when shit hits the fan.

Only incompetent people wouldn't blame the IT manager for choosing Oracle.

The answer is of course no.

The Oracle pitch is "We do this for Michigan State and can do X, Y and Z". That is true.

Oracle gets alot of shit for this stuff because they own the market for enterprise financial software. Their software is gross, they bought out the competition and generally suck, but that is what the market demands.

My uncle is a locomotive engineer. His railroad makes a conscious decision to remove any creature comforts from the cab, like a chair cushion. It's needlessly uncomfortable and is awful, but that doesn't mean that GE (or whomever) sucks. They delivered what they are asked to do.

One of the setups I come to almost expect in maritime accident reports is an officer taking a night watch, alone, while somewhat fatigued. Warm, dark, quiet. After a few hours the ship has drifted significantly off its intended course. For unknown reasons (the officer was probably asleep) no attempt is made to correct... When they do wake up, they often don't have good situational awareness, and their focus is on pretending nothing is wrong, even if they are in fact already in serious trouble and need all the help they can get.

Night-time railway accidents are less common in my country, perhaps because the relevant unions insist on proper "if you're too tired just go home you aren't fired" type rules to reduce fatigue errors, but I do recall a case where a freight driver was dipping in and out of consciousness. When he was conscious he could see a red lamp out of the window, signalling "Danger" ahead so he'd reduce the traction power. If he'd been less sleepy he'd have noticed those red lamps were not getting closer each time, they were moving gradually away from him. His train was on a steep climb, and he'd reduced power so much it was gently rolling backwards. Oops. Nobody died, that time.

WTF are the administrators paid for if not that? Isn't that literally their whole job?

The school cannot handle the requirements or implementation. Of course, but maybe a smaller, more focused provider than oracle can?

And yes, you have to be at peace with your data being god-know-where.

You can reference the Texas power outages. Just because the requirements were to fail in cold weather, doesn't mean it was a good choice

I wound up going back several years later when I was thinking of a different field and they had that trashcan CUNYfirst system. By then the UI was already outdated and clunky. I wound up with the wrong class the first time around because of some issue and have to have it changed post payment which was time consuming. I would have rather used the IBM's.

I used a class registration system that wasn't this and didn't mention a "shopping cart" metaphor anywhere, but the system was exactly the same. Why object to calling it a "shopping cart?

I can only imagine how hard it was to populate the available classes. I wouldn't be surprised if part of the huge administrative bloat in the various departments was people who's only job was to force data into and out of PeopleSoft.

There were a few dozen of us spread across the UW System who's sole job it was to extract and massage data from these databases... One, very large, central database that handled data for UW and every satellite college they had.

It's lazy UI but in some ways it's going to be more intuitive, not less, to any student who's ever used an online retailer before.

I once was deposed in a lawsuit regarding Oracle, and part of it was basically a UI issue. We had a paid customized interface that allowed users to upload basically a blob text or document (word, etc). When we went to UAT, we didn't see it in the UI in the app itself (peoplesoft):

Us: "Okay, we can upload stuff, where do we find it?"

Oracle: "Oh, that wasn't part of the project"

Us: "Yes it was, see spec #<insert #>."

Oracle: "... Well, it's in the database. A DBA can access it, so we met the spec."

Us: "That is not a rational interpretation of '... and users will access it within the application...' "

Oracle: "We're happy to build that customization for another <insert massive $$$ sum>"

This was only one of a few material issue that led to the lawsuit. We probably wouldn't have sued over it alone, which should give you a hint at just how bad some of the other problems were.

As an aside, the $/hour billed for customizations was $600, but the work was also outsourced to India. I asked a PM from Oracle how this was justified, and they gave a vague answer about multiple layers of management needed to oversee the developers. I pointed out that this price tag would allow for 3 managers per developer, with each person in the chain making ~$200k/year, and still allow Oracle $200/hour profit. I got a ¯\_(ツ)_/¯ response. That I have to blame on the organization & not Oracle though. I had the detailed ~500+ page RFP response & implementation plan, which specified 1,000 hours of custom work. But I didn't have the actual contract to see what additional work would cost. The organization either didn't negotiate that in advance (so their fault) or they knew in advance & agreed to it (also their fault)

The thing was, when it rolled out (2007), everyone was thrilled with it because it replaced a mainframe that you had to telnet into. That mainframe had hours of operation, i.e. you couldn't look at the course catalog at night because they shut down the mainframe to save power.

Oracle makes a ton of money because what they're competing with is just incredibly terrible.

This was last year btw.

Things are a little better these days where you can often find apps (usually SaaS) that fill a specific need very well, but then you're usually stuck managing a dozen different products and a massively more difficult integration between all of them.

If you're using Oracle to power a home-grown system though, yeah: With a little effort you can swap the backend for something where the vendor won't show up with a surprise multi-$million invoice along with pre-made lawsuit papers if you don't sign it.

What really happens is entirely different. Oracle takes the request, foots the bill for an army of contractors to build it, if they haven’t already, and pass it off as we had it the whole time, just like we said.

Then up charge. Shake down street. Lock in.

I’m sure not all of Oracle operates this way, but that’s been my experience.

Homegrown systems are an easy low-hanging fruit to cost reduction.

Software to tackle the BPaaS on top of an Oracle add-on service, requires a pitch deck.

That must be what happened in my case! I didn't realize it was a standard practice. A large part of the lawsuit revolved around functionality the RFP required, Oracle said they had, and that Oracle even showed in what may have been faked screenshots during our review of vendors. Only in our case, the army of developers never actually finished it and Oracle came back to us with another massive bill to complete the project.

I'm not sure I've ever heard someone who's worked with Oracle who didn't have a bad experience. But if you have and you're reading this, just drop a quick comment to say "yeah, they were okay"

There was a complaining faction in my org, mostly disgruntled hold-outs from before my tenure. Whatever complaint they thought they had, I could bring it to our account manager, and it was pretty easy to see the root of the problem came from personal issues in staff, and insecurity with changing to better technology that they were not experts in.

Now that I've got mostly oracle certified engineers and oracle familiar managers on board I don't hear any complaints. It's a great product, and in all my future roles, transitioning to Oracle is one of the first things I'll do when I get on board.

Oracle as a product company (applies to all Enterprise Product companies) will not charge separately for a customization.

In any case, these were from the the actual consulting arm of Oracle inc. And IIRC, our Associate VP for IT said we shouldn't use both Oracle & Oracle Consulting, but that person was overruled by the VP for IT. I the VP was kept around until the lawsuit settled because it might have been taken as a sign or admission of culpability to fire the VP before that, but shortly after the settlement (which we won, incidentally) that person moved on for "other opportunities".

From what your describing it seams that what was missing was an API allowing browsers to get back the uploaded files?

Was that project worth it? We didn't have much choice. Our old ERP was close to 35 years old, and no longer supported, so we were paying large sums of money just to get annual compliance updates for state & federal requirement. The new system is fine, does what it needs to do, but it's a monolithic ERP system: It does just about everything, but it's mediocre at all of it with most of it's tech perennially 4 to 7 years out of date with the best of breed. So, in the areas where we really need best of breed, we bought that and integrated it into the ERP, which is considered the "system of record". So it's fine, it works, I guess. The real kicker though is that the system we ultimately chose, while not an Oracle product, still required an Oracle DB. Fortunately we don't actually deal with Oracle as a vendor though. Our vendor is basically a VAR.

Are you sure they wouldn't like a new interface if it was actually a real improvement?

I'm reminded of a story[0] about Norwegian doctors refusing to use the new GUI system and stubbornly continuing to use the obsolete text-based system. Were they luddites? No. The old system let them quickly navigate with the keyboard in seconds while talking to the patient. The new system required them to use the mouse, so they had to spend a lot of time focussing on using the GUI, rather than helping the patient.

[0] https://news.ycombinator.com/item?id=10289172

Sometimes, a core job of IT is eating all the pain of using the old technology that the users already know so they can go do their job and the company can achieve its objectives.

The terminal apps, at least, have some history behind them and properly trained folks can be very fast with them.

But we're talking here about shitty early Swing-apps (or Oracle forms). If you've ever used one it can be incredibly frustrating. Moreover, people are usually just tossed into a role where they have to use these M-F-ing interfaces with no training. Who makes a table with cells too small to fit their content and where you can't resize the table? Oracle. That's who. And they're paid for it like you wouldn't believe!

To be fair, lots of software of those years were like this. SAP Ariba was just like this, and so were old Microsoft products

Encountered a similar issue many years ago at a company trying to sell a new ad booking system to Titan - they had crufty old VT100 based systems, the new one was fancy AJAX HTML whizzbangs. Exactly the same result - they could handle 5 bookings on the old system in the time it took to do one on the new (which was also terrible and ugly besides slow.)

Here is the thing, the reason these people would want to keep using the software is... The replacement will be broken and they'll spend years with bugfixes and what not to get to the exact same position they're in just now.

Using old UIs isn't fun but if it works it's good enough. This is a case of everyone getting used to the workarounds until no one knew about the workarounds.

Somebody requested that feature, and then someone discovered that if you hack around your system and set precise combination of fields, it will do the trick. So man hours were saved, some memo added to docs, and then the knowledge was lost.

And all of that instead of adding separate form for specific case. I've seen it more than once.

Everybody who was there when the software was created is long gone. It hasn’t been updated in decades. Nobody really knows how it works, and training is entirely folk wisdom and cargo cult practices handed down over generations.

The people that did know how to use it got constant phone calls (yeah, PHONE CALLS, because that's how these people roll) to do ad-hoc queries for management. I guess I can only fault them so far. This skill put food on their table and kept them employed even though their end product was endless dowdy bar-charts and badly formatted tables.

Now we're using a new product. It's a "BI" tool.

So many corps are doomed to repeat the bad parts of computer application history over and over and over again. In the case of "BI" or "Business Intelligence", it got it's start when middle-management got tired of begging for reports that weren't baked-in by the application vendor years ago. They started with getting permission to directly query the database (a profound coup by their standards). When they found that it was _actually_ possible to get useful information from the database, it looked like "intelligence" to them-- hence the term "business intelligence".

Oracle will audit them and find that someone accidentally enabled an extra, unused feature during an install one time. So, there goes another 10 million.

I have a soft spot for Citi because where else can you get 2% back on your credit card? They're not bad, they just like giving money away to their customers...

There's a psychological defect in the human brain called Anchoring. If you are first presented with the wrong answer to a problem, it affects your ability to objectively come up with the 'right' answer. You will not catch all of the garbage I threw at you because I threw it. "Double" checking is a misnomer because it's not really doubling the number of brains.

Some things should work more like nuclear launch codes. I don't approve your action, I perform the exact same action, and if we both did exactly the same thing, then something happens. If we don't, then nothing happens or you get an error.

Might this problem still have happened? Yes. But the first time it happened would probably have been taken more seriously, instead of what we usually do: deflect, deflect, deflect.

Doctor 1 gathers evidence and makes a preliminary diagnosis and writes it down. Doctor 1 then presents only the evidence to Doctor 2. Doctor 2 comes up with an independent diagnosis and writes it down. They then see if they match.

I mean, it's better than cutting off the wrong leg, but boy is it rough.

Unfortunately because of the court case, instead, people associated are the users.

Pretty much every modern financial app I've used has one of those confirmations - some better than others but all pretty clearly state "you are going to transfer $XXXX.XX to ACCOUNT_YYYY -- click here to approve transfer".

Even venmo's quick prompt to make sure you really mean to send your pet sitter $125.00 before initiating a transfer would have avoided this problem.

The linux approach is "what you are doing is not my problem, so go do whatever you want".

For a bank, what their employees do is very much their problem.

Well, no, it's not like that at all. There IS a confirmation window; it's just not embedded in the software. Three different people -- the flunky, his boss, and his boss's boss -- were all required to confirm the validity of the transaction before making it.

> Raj then proceeded with the final steps to approve the transfers, which prompted a warning on his computer screen — referred to as a “stop sign” — stating: “Account used is Wire Account and Funds will be sent out of the bank. Do you want to continue?” But “[t]he ‘stop sign’ did not indicate the amount that would be ‘sent out of the bank,’ or whether it constituted an amount equal to the intended interest payment, an amount equal to the outstanding principal on the loan, or a total of both.”

(from the judge's opinion, via Matt Levine and Bloomberg)

In this case it may be the best option you have, but the UX you should be aiming for is always undo.

Humans are always sure this is what they wanted to do right up until they understand the consequences. Then they experience regret. If possible design your software so that regret has a natural response in the interface in the form of an "Undo" option.

A confirmation step doesn't trigger that "understand the consequences" outcome, and so users will mostly be annoyed by the confirmation, and pick "Confirm" even when in fact they'll immediately regret that once they do it. They may even ask you to add another "Confirm" step. You want "Undo".

Notice the law here reflects this preference for Undo. If Citibank wired $500M to me the law says they get to Undo that, because they didn't owe me $500M and so that's just a mistake. They can't undo this because it looks exactly like a legitimate payment to a creditor, and if you could always undo those it opens a real Pandora's box. Normally you can "undo" paying a willing creditor because they'll just lend you the money again. But of course not everybody is a willing creditor, as in this case. Boo hoo for Citibank.

"Linux gives you enough rope to shoot yourself in the foot. If you didn't think you could shoot yourself with a rope, you should read the man page first."

That said, as Linux has gotten more popular, it's put more safeguards in. For instance, you can no longer "rm -rf / file.txt".

I know someone that works for a (smaller) bank regularly processing loans in the mid six figure range. When I think about all the steps she has to go through around verification on each loan, It's just crazy to me how they're able to accidentally send an amount over 1000x times what she has to deal with.

It simply should not be possible, this is about literal tons of money, they shouldn't rely on something this banal.

Simple safeguards can be put in place to prevent this. They don't even need to be very fancy:

Fairly sure a simple trained-human-readable description of what was about to happen (think: this money will go to this acct, this money here and this other money over there) would have saved this people a whole lot of grief.

I know you didn't mean it this literally but the issue is worse than that. They weren't confused. If they were confused, they would have sought clarification. Instead, they were sure they were doing the right thing.

  Overwrite default settlement instruction

  Select the ‘Overwrite default settlement instruction’ check box to confirm
  that the liquidation entries should be posted into the Internal GL account. 
  System posts the entries to the Internal GLs only if you check this check box.
  Otherwise, system posts the entries as per the settlement instructions specified
  for the component of the contract.

I doubt Citi would want to pursue that sort of a suit, since it would lkely highlight they were at fault. Sure the software is probably difficult to use, but if they already knew how to use it, then using it incorrectly is not the fault of the UI.