News for hackers. Stuff that matters.
Re:How do bank payments actually work? by jrcii on Wednesday January 20, 2016 @10:03AM (#10938496)
I've asked bankers at the branch about this before and received a vague answer about how they need time to verify things, but I strongly suspect they were just non-technical and didn't know.
Re:How do bank payments actually work? by ChuckMcM on Wednesday January 20, 2016 @12:06PM (#10939504)
I don't know how current it is but back in the 90's when I was dealing with technical issues with payments at UBS (UBS was a big Sun purchaser) one of their bankers described bank transfers as a giant exercise in "liability hot potato". Which is to say that at each step of the way the person receiving the funds was trying to avoid becoming liable for them until they were really certain they had them in an account somewhere, and people sending funds wanted to be absolutely sure the destination had accepted liability for the funds before they were willing to release them. Because when it went wrong, they ended up liable for funds they no longer had. And he suggested that there were people that tried to make that true (basically stealing from the system) and that it happened enough to be priced into the transaction fees.

The amount of money that was flowing on a daily business through the bank was staggering.

Re:How do bank payments actually work? by jrcii on Wednesday January 20, 2016 @12:19PM (#10939608)
Given the crowd on HN I'm surprised there isn't at least one person chiming in that works on these backend bank systems.
Re:How do bank payments actually work? by tolle on Wednesday January 20, 2016 @01:17PM (#10940114)
A wild guess would be that roughly 90% of all systems handling this are written in COBOL running on z/OS or IBM i. And that most of the developers are currently close to 60, spending their free time with their grandchildren and not on Hacker News.

I'm not much help either since I work with mortgages, and having it all explained during a lunch break might be a bit much. There are also, depending on company and country, rules and laws about what you can talk about.

Re:How do bank payments actually work? by zhte415 on Thursday January 21, 2016 @07:03AM (#10944660)
There are still a lot of developers for these systems. Updates happen constantly and developers are constantly in need and training-up.

For example, credit card legal and security compliance updates happen a lot with really strict deadlines that have to be adhered to, or else the credit card issuer gets into hot water.

Some organisations choose to merge old features that differ by country get merged in a central trunk with flagable switches. The code bases are very active.

My company (not going to say the name here) employ 500+ mainframe engineers, some working on new projects and codebases because mainframe is the preferred platform.

If you or anyone else would like to get in contact regarding COBOL or Mainframe in general, especially getting work done, especially in finance, please find my email in my profile.

Re:How do bank payments actually work? by jrcii on Thursday January 21, 2016 @05:02PM (#10948800)
> There are still a lot of developers for these systems.

I hear Initech is hiring.

Re:How do bank payments actually work? by PeterisP on Wednesday January 20, 2016 @03:25PM (#10941062)
I've worked on those systems and there's not much more to say, the hot potato analogy is pretty much a perfect description.
Re:How do bank payments actually work? by jrcii on Wednesday January 20, 2016 @04:31PM (#10941525)
What protocols and software stacks do they use? How are they networked? What are the steps in the verification process? What prevents it from being able to work in real time? Where is this type of information formally recorded? I had a hard time turning it up. For such a critical and prolific infrastructure it's unusually opaque.
Re:How do bank payments actually work? by rconti on Wednesday January 20, 2016 @07:30PM (#10942535)
I once worked with an ACH system where the transactions were in a txt file on a Windows box that was, IIRC, copied over ZMODEM after a dialup modem connection was opened to whatever the other party was (bank? processor? I don't know). Obviously, given the liability involved, I never messed around with it. But I always wondered what would happen if some amount was different than expected. One would hope there's all kinds of verification going on, but I kind of doubt it.

This was only 10 years ago.

Re:How do bank payments actually work? by otakucode on Wednesday January 20, 2016 @11:09PM (#10943295)
When I worked at a data center for a bank, we had basically the exact setup you are describing. Part of my responsibilities was to take note of the amount that came through via ACH on that Windows machine (Windows 3.11 actually... this was in 2001) and enter it into a Lotus 1-2-3 spreadsheet, along with totals which I got from reports that ran after the days batch processing completed, and make sure they balanced. If they didn't balance, I had to go hunting and pray that I simply transposed a number or typoed something along the way... if I did not, I honestly don't know what happened. I had to call one of the other employees to come in and orchestrate getting it fixed. (I was alone at the data center, which consisted of a mainframe and a handful of PCs... probably an order of magnitude smaller than what you would picture if I just say 'data center') Luckily, it happened very very rarely. Maybe a handful of times in the 4 years I worked there.
Re:How do bank payments actually work? by PeterisP on Wednesday January 20, 2016 @05:57PM (#10942048)
> What protocols and software stacks do they use?

Heterogenous, differs between institutions, differs within institutions, also many/most institutions are a mix of incompatible systems because the business institutions are a merger of mergers with acquisitions of mergers, all while requiring to keep constant operations, full archives of all past deals of all those legacy institutions and compatibility.

In some spaces, Java is popular, in some spaces COBOL is used, and many have something from a multitude of different exotic niche platforms. In any case you'd expect to have a mix of systems/platforms, some of them modern, some of them older than you are. You'd have 'common' platforms/stacks e.g. oracle on linux, or mssql on windows, you might encounter something also running on LAMP somewhere, and you might have e.g. a custom platform with integrated DB-like and programming-like functionality running on a particular version of AIX, all of those in a single institution.

> How are they networked?

All kinds of things. Some use common TLS channels. Some parts of business may even use simple file transfer or public email without expecting the channel to provide any security, but rely on each separate message being securely encrypted and signed. For anyone that you "don't know personally", there's SWIFT that will give you secure though not cheap messaging - if you get a swift message from e.g. FBNINGLA then you can be sure that yes, this actually is an authorised representative of the First Bank of Nigeria.

> What are the steps in the verification process?

Depends on your pre-existing relationship - if you trust them up to some limit (equivalent to a credit line up to that limit), then you do technical verification that the request is authorised and authentic and that's it. If you don't, you wait until they somehow hand over the 'cover funds' to someplace that you securely control, verify that it is received there (e.g. account statement from that place at the end of their business day), and then proceed further.

> What prevents it from being able to work in real time?

I believe mainly because multiple independent parts of the process that have no motivation to change, and often have motivation to not change - if you agree to do your part of the things faster and cheaper, it adds cost and decreases fee revenue; and changing things requires cooperation and investment from a majority of players.

Up to some part, there may be legal obstacles that prevent some particular ways to be done faster (the liability 'hot potato' issues) but IMHO those would be quickly solved by the industry, lobbyists and involved lawmakers if the motivation was there.

> Where is this type of information formally recorded?

It's heterogenous, so it's a multitude of similar-and-different things. For any particular payment system you'll have short public descriptions and also available training courses, technical standards, etc though often at a significant fee. It is niche expertise required within that niche, and not particularly PR-useful to publicize in detail, probably entirely the other way around.

Re:How do bank payments actually work? by choosername on Wednesday January 20, 2016 @05:13PM (#10941782)
potatoes hardly stay hot that long, isn't it
Re:How do bank payments actually work? by needusername on Thursday January 21, 2016 @02:03AM (#10943796)
Huh? Would a move to a real time gross settlement system fix all of this? We run all interbank transactions over a real time gross settlement system. There is no counter party risk. Everything is backed by a giro account at the central bank.
Re:How do bank payments actually work? by PeterisP on Wednesday January 20, 2016 @03:23PM (#10941047)
The "need to verify" is not a technical need and can't be solved with technical means - it's all about counterparty risk, it is a business need to verify so that you're not vulnerable to all kinds of exploits and fluctuations.

It is trivial to have the payment messages happen in near-real time. However, to actually make a final credit in the receiving account and allow the end user free access to withdraw that money, you need one of three things:

1) the receiving institution simply trusts that they'll receive the money and releases the money as soon as they can - which is not realistic due to counterparty risk, fraud/revocation risk, etc and can happen only either if the whole payment happens in a single central system/institution (e.g. something like paypal) or the system includes very high fees to cover those risks.

2) each sending institution has a large binding collateral and there are appropriate safeguards that in the case of someone "going bad" the receivers are going to get paid anyway - which requires a strong central system to manage it, and the institutions deffering a lot of power and money to it (in this context, a million doesn't come close to a lot of money). This is the system described in the original article - the payment gets credited/finished first, and covered/settled between the banks afterwards.

3) in the absence of strong guarantees or blind trust, everyone in a payment chain pays the money forward only after they have been actually paid to cover the payment, adding days and days to the processing time. This is the current US system and also how general arbitrary international payments can be settled - particularly in international payments, you can get the full payment data nearly immediately, but you generally wouldn't receive the funds until every institution involved has covered their risks.

Re:How do bank payments actually work? by superuser2 on Wednesday January 20, 2016 @04:14PM (#10941413)
These are real problems. However they are solved elegantly by central banking, and indeed that is what we have in the US: ACH and FedWire. Each bank holds an account with the Federal Reserve, and can make cheap instantaneous or free nightly batch transfers between its Fed account and other banks' Fed accounts. The strong central system extracting guarantees and ensuring minimal abuse is a major production, but we still use it for all electronic transfers.

The problem is that many American banks also process their own transfers in a nightly batch on the same 60s mainframe application they've been using since mainframes were cool.

First business night: money is taken out of your account.

Second business night: money is moved between banks via the Fed.

Third business night: money from the Fed is credited to recipient account.

If both banks were to actually do online transaction processing (like MC/Visa) then you'd only wait for the Fed's nightly batch. Some institutions like Alliant Credit Union have the rare differentiator of actually transacting with you and the Fed on the same day, instead of in separate nightly batches.

FedWire is actually instantaneous settlement, but the banks have all apparently conspired to charge an enormous markup on it. It doesn't help that cheap ubiquitous wires would outright destroy MC/Visa, who effectively get to tax all consumer spending by being the only system with live/online transaction processing for authorizations.

Re:How do bank payments actually work? by PeterisP on Wednesday January 20, 2016 @07:34PM (#10942552)
The EU solution to a very similar problem that we had earlier was essentially a single item in legislation (the payment services directive) that states that if a consumers payment is not credited to the beneficiary within a specific time limit, then the consumer is entitled to compensation from their (sending) institution. The industry had a couple of years before it came into force, by that time everybody implemented it.

I believe that a similar approach is the only way that can be effective in achieving a similar result in USA - unless some regulator draws a line and says "your product must meet a certain quality level or it's not fit to be sold" the current market has no reason to change, it's more profitable as it is now.

Re:How do bank payments actually work? by mikaelm on Thursday January 21, 2016 @03:35AM (#10944072)
Hey PeterisP, what does the EU payment servies directive specify as max send time and for what cases, can you give more exact reference? Thanks.
Re:How do bank payments actually work? by PeterisP on Thursday January 21, 2016 @07:49AM (#10944842)
The key point for this is #69 of http://eur-lex.europa.eu/legal-content/EN/ALL/?uri=CELEX:320... stating that they must be credited to the beneficiary "at the latest by the end of the next business day".

+1 business day if made in paper, and cut-off times apply - if you make an order at late evening, the bank can treat it as received in the following business day.

"for what cases" is a bit tricky. It must apply to all intra-EU deals in EUR and for the EU countries that don't use EUR, their national transactions. But for other combinations of countries and currencies (e.g. to non-EU places like Switzerland, local transfers in USD, etc), it varies, and there's some room for interpretation by member states.

Re:How do bank payments actually work? by renaudg on Wednesday January 20, 2016 @08:58PM (#10942858)
>It doesn't help that cheap ubiquitous wires would outright destroy MC/Visa, who effectively get to tax all consumer spending by being the only system with live/online transaction processing for authorizations.

Cheap (actually: free) ubiquitous wires are a reality in the UK as described or in the Euro zone, and I think MC/Visa do just fine there.

Re:How do bank payments actually work? by pbhjpbhj on Wednesday January 20, 2016 @04:05PM (#10941335)
>It is trivial to have the payment messages happen in near-real time. [...] receiving institution simply trusts that they'll receive the money and releases the money as soon as they can //

Surely the message is "the money", it's not like a truck pulls up at the bank each night with the transaction balance for the day?

I don't see how there's a risk if there's a transactional process: for example Bank A sends request for money from B (eg to fulfill a payment by debit card); B reserves the money and sends a verified payment (crypto signed and what-have-you); A processes the payment and increments the relevant account with reserved funds and sends a receipt to B; B processes the receipt completes the decrease on the relevant account using the previously reserved fund and acknowledges the receipt; A receives the acknowledgement and verifies the funds as available in the account.

So, if B doesn't have funds to reserve then the payment can't be processed and A won't acknowledge any receipts claiming the transfer of monies. "A" thus won't increase a balance for any accounts. Where's the risk?

Re:How do bank payments actually work? by superuser2 on Wednesday January 20, 2016 @04:16PM (#10941423)
"The money" is money held in the bank's account at the Federal Reserve, and sending an electronic message to the Fed to transfer this money immediately (FedWire) counts as actually moving money. The only problem is that all banks have decided to charge $25 for this service, despite it costing them on the order of $1.
Re:How do bank payments actually work? by pbhjpbhj on Wednesday January 20, 2016 @04:26PM (#10941485)
But they have agreements with banks to process the final transaction at the close of day [don't they??] - so the reserve would then charge them $1 per day per bank they had to settle a debt with? Presumably the transaction receipts and acknowledgements are binding meaning they can't legally screw another bank out of that money. So the risk is that they'll have to sue another bank in to the ground and make oodles of money if that bank refuses to settle with them at the end of the day? [Off-shore is obviously going to be different.]

Can someone speak to the situation in the UK in this respect too, please.

Re:How do bank payments actually work? by PeterisP on Wednesday January 20, 2016 @06:08PM (#10942134)
Insolvency screws another bank out of that money no matter what legally binding acknowledgements you have, and you won't be able to recover much from suing because, well, they don't have enough to pay.

They are rare, but not that rare, especially whenever the economic cycle again and again reaches some hard times. for example http://www.statista.com/statistics/276231/bank-insolvencies-... lists a few hundred insolvent institutions just in the last dozen years.

Re:How do bank payments actually work? by PeterisP on Wednesday January 20, 2016 @05:28PM (#10941875)
First, let's not look at credit card payments which work very differently from what you describe (the 'network' i.e. MasterCard company, Visa, etc play a big role) but at payer-initiated transfers.

Second, keep in mind that in almost all cases you won't have a bilateral relationship, this is unrealistic to have every institution to handle close relationships with each other - instead, usually the best case you can rely on is a "star" scheme where any A in some market can reach any B in the same market by sending a payment message + money to some "central" C and asking that C to forward the same money to B; and in more distant situations you can easily get 2,3,4 or more intermediaries for various reasons.

Third, be very careful with terms like "the account" - unless you add more specifics, all that "the account" generally means is "our accounting department believes that this is how much you owe us"; it should reflect 'having money' if accurate but it is not the same e.g. if you're unable to collect what your accounts say you own. If you say "A has $10 in their account with me" it literally means "As far as I know&have accounted for, I owe $10 to A, and should allow them to do stuff worth $10 as they order".

But for the main points - your proposal has two weak spots. 1) It requires any paying institution to have large available reserve funds with all receiving institutions. I.e. if I'm bank A, and I want my large corporate customer to be able to send $100m to any of the banks B,C,D,E,F,G,H,I,J,K when chey choose to, then I would have to keep $100m 'in my account' with each of them (or alternatively, have a credit line, where I'm sure that they will allow me to overdraft $100m) - in this case this requires having $1b of capital just so I can have this capability even in this small toy example. If you'd want to have more potential recipients instead of 10, you'd need even more capital. Part of this is actually done so in practice, but as you can see it's not that simple.

2) Your proposed messages (which are similar to real world implementation) + appropriate legal agreements only achieve that after all this ritual is completed, there is a legally binding situation that yes, the sending bank does owe the receiving/executing bank the required amount.

However, from a risk-averse viewpoint "the money" is transferred from A to B if and only if B is ensured to have full access to the full money (as in, they can actually request it driven by a truck to them and be sure that this will happen) even if e.g. A declares insolvency after sending all the required messages.

Do note that in the usual insolvency procedures, if A has $10 as "funds available in the account of A" (i.e. you owe $10 to A) and also A legally owes you $10 for the payment you did an hour ago, then you will not be able to take that $10 - all their assets (e.g. money in that account) will be split among all the creditors, so you can expect to actually not get most of that money in such an event. This is part of the 'liability hot potato'; it goes a bit deeper but you'd need a finance lawyer to explain it in more detail.

You would expect to be safe from counterparty risk only after 'your account' with the sending bank shows 0, they have legally acknowledged so (which for various legal reasons depending on country may require an end of business day, not just a signed message from them), and you have "got the money" in some form that's more trustworthy than simply 100% legally&cryptographically secure acknowledgement that A owes you the money - e.g. a message from a truly trusted institution (e.g. your central bank) that they owe you this money and it will stay so even if A goes insolvent.

Re:How do bank payments actually work? by pc86 on Wednesday January 20, 2016 @10:07AM (#10938525)
Is part of the problem inertia? What I mean is if there are multiple pending transactions on an account, is it even possible from a technical perspective to say "Is there $100 available in this account for immediate withdrawal?" I would suspect it is possible, but I also wouldn't expect the majority of US banking infrastructure to run on AS400s.
Re:How do bank payments actually work? by icebraining on Wednesday January 20, 2016 @01:43PM (#10940316)
Isn't that the available balance? At least here in Portugal, we have three balances: accounting (based on all non-pending transactions), authorized (accounting plus all pending transactions) and available (like authorized, but just with the pending debits). The latter is the amount actually "yours", which you can use without paying interest on.
Re:How do bank payments actually work? by pc86 on Wednesday January 20, 2016 @01:53PM (#10940388)
Yes but particularly for restaurants where us Americans will tip 99% of the time, the "pending" amount is often different from the finalized amount. You won't usually see that until the item actually posts, which could be several days.