I can't disclose much due to NDA, but what basically occurred was, the tool used to enforce policies across the enterprise was accidentally made to reboot everything. Simple as that.
Re:Fed Is Down
by SQL2219 on Wednesday February 24, 2021 @02:20PM
(#26254316)
February 24, 2021 2:17 PM Eastern Time
The issue impacting Central Bank applications has been resolved and users may resume normal access.
February 24, 2021 2:14 PM Eastern Time
Beginning around 11:15 a.m. ET, Federal Reserve Bank staff became aware of a disruption for all services. Our technical teams have determined that the cause is a Federal Reserve operational error. We will provide updates via service status as more information becomes available. We acknowledge that payment deadlines are impacted and will communicate remediation efforts to our customers when available. Thank you for your patience while we work to resolve the issue.
February 24, 2021 1:18 PM Eastern Time
The Federal Reserve Bank staff is currently investigating a disruption to multiple services. We will continue to provide updates as soon as they are available.
February 24, 2021 12:43 PM Eastern Time
The Federal Reserve Bank staff is currently investigating a possible issue or disruption to multiple services. We will provide updates as soon as they are available.
Re:Fed Is Down
by chewz on Wednesday February 24, 2021 @02:48PM
(#26254713)
February 24, 2021 2:46 PM Eastern Time
The Fedwire Funds Service, Fedwire Securities Service, and National Settlement Service have resumed processing and are operating normally. The Federal Reserve Banks have taken steps to help ensure the resilience of the Fedwire and NSS applications, including recovery to the point of failure. Nevertheless, it is critical that you reconcile the messages you believe you had sent or received through the services after waiting several minutes for the backlog of messages to finish processing before you submit additional messages to the Fedwire Funds Service and Fedwire Securities Service and additional settlement files to the National Settlement Service.
February 24, 2021 2:36 PM Eastern Time
We are in process of restarting the Fedwire Services and National Settlement Service and expect to resume normal processing this afternoon. We plan to extend the cutoff times for the Fedwire Securities Service and may also extend the cutoff times for the National Settlement Service and Fedwire Funds Service. We will provide additional updates about these extensions.
February 24, 2021 2:32 PM Eastern Time
Access to FedCash Services via the FedLine Web solution has been restored. We are extending the cash order deadline for FedCash Services to 3:30 p.m. local time of the servicing Federal Reserve Bank. Please contact your local FedCash Services customer support if you have any questions. For a complete list of customer support contacts for your institution, please visit the Contact page.
Re:Fed Is Down
by ilaksh on Wednesday February 24, 2021 @02:37PM
(#26254555)
See the response to my comment below. My impression is that the best solution is actually a high-thoroughput cryptocurrency for settlement.
But cryptocurrencies are philosophically opposed to central control. And authorities are generally behind the times when it comes to technology. But it boils down to outdated technology based on "if it ain't broke, don't fix it."
Re:Fed Is Down
by toomuchtodo on Wednesday February 24, 2021 @02:16PM
(#26254258)
This is a demonstration that all systems have outages. It does not demonstrate a distributed ledger system would be superior. Distributed technologies have different trade offs and failure modes.
Re:Fed Is Down
by toomuchtodo on Wednesday February 24, 2021 @02:19PM
(#26254303)
It's going to be quite some time before the public at large trusts crypto over a centralized finance system backed by the full faith and trust in the US government (if ever). Currencies aren't about technology, they are an expression of power and trust.
Less than two hours of downtime is nothing to write home about, and the Fed is down less frequently than major cloud providers and others those in technology would be familiar with and consider critical services. Simplicity is the ultimate sophistication, and it doesn't get much more simple than message queues and worker processes.
Re:Fed Is Down
by ilaksh on Wednesday February 24, 2021 @04:53PM
(#26256556)
I was suggesting using cryptocurrency (such as Ethereum or PolkaDOT) to implement the centrally-controlled functions of the Federal Reserve currently being handled by simplistic queues or whatever it is.
So this is a different discussion than the one your previous comment argues against.
"Simplicity is the ultimate sophistication" is a nice-sounding catchphrase but in many contexts does not hold water. For example, systems without redundancy are objectively simpler but not more reliable.
But more fundamentally, currency and money IS a technology. It's use is as you say to distribute power and express trust. But fundamentally it is a technology used for that purpose.
Cryptocurrency is ultimately about putting power in the hands of the people and trust in mathematics rather than central authorities.
Eventually the central banks and banking institutions as they exist now will be phased out and replaced with different paradigms that have some similar functions but more sophisticated technology.
Re:Fed Is Down
by farisjarrah on Wednesday February 24, 2021 @06:04PM
(#26257388)
I am not for or against cryptocurrencies, and not a cryptography expert, so take what I say with a grain of salt.
You don't necessarily need to use a publicly available cryptocurrency that exists in the wild today. If an organization was motivated enough and thought it was a good idea, they could simply launch their own private blockchain on their own hardware that they control. With your own blockchain on your own hardware you could still reap the benefits of decentralized compute power, and build your own smart contracts system that can broker chargebacks. The barrier to entry for an endeavor like this is the lowest its ever been too, you can even spin up a managed blockchain on AWS these days.
February 24, 2021 2:14 PM Eastern Time Beginning around 11:15 a.m. ET, Federal Reserve Bank staff became aware of a disruption for all services. Our technical teams have determined that the cause is a Federal Reserve operational error. We will provide updates via service status as more information becomes available. We acknowledge that payment deadlines are impacted and will communicate remediation efforts to our customers when available. Thank you for your patience while we work to resolve the issue.
February 24, 2021 1:18 PM Eastern Time The Federal Reserve Bank staff is currently investigating a disruption to multiple services. We will continue to provide updates as soon as they are available.
February 24, 2021 12:43 PM Eastern Time The Federal Reserve Bank staff is currently investigating a possible issue or disruption to multiple services. We will provide updates as soon as they are available.
February 24, 2021 2:36 PM Eastern Time We are in process of restarting the Fedwire Services and National Settlement Service and expect to resume normal processing this afternoon. We plan to extend the cutoff times for the Fedwire Securities Service and may also extend the cutoff times for the National Settlement Service and Fedwire Funds Service. We will provide additional updates about these extensions.
February 24, 2021 2:32 PM Eastern Time Access to FedCash Services via the FedLine Web solution has been restored. We are extending the cash order deadline for FedCash Services to 3:30 p.m. local time of the servicing Federal Reserve Bank. Please contact your local FedCash Services customer support if you have any questions. For a complete list of customer support contacts for your institution, please visit the Contact page.
I know the Fed is responsible for resolving transactions between banks, but one would expect they'd have a very very broad and resilient system.
But cryptocurrencies are philosophically opposed to central control. And authorities are generally behind the times when it comes to technology. But it boils down to outdated technology based on "if it ain't broke, don't fix it."
Seems like a good demonstration of why a distributed technology like a "real" central bank digital currency would be an improvement.
And some newer cryptocurrency strategies have very high transaction rates.
Less than two hours of downtime is nothing to write home about, and the Fed is down less frequently than major cloud providers and others those in technology would be familiar with and consider critical services. Simplicity is the ultimate sophistication, and it doesn't get much more simple than message queues and worker processes.
So this is a different discussion than the one your previous comment argues against.
"Simplicity is the ultimate sophistication" is a nice-sounding catchphrase but in many contexts does not hold water. For example, systems without redundancy are objectively simpler but not more reliable.
But more fundamentally, currency and money IS a technology. It's use is as you say to distribute power and express trust. But fundamentally it is a technology used for that purpose.
Cryptocurrency is ultimately about putting power in the hands of the people and trust in mathematics rather than central authorities.
Eventually the central banks and banking institutions as they exist now will be phased out and replaced with different paradigms that have some similar functions but more sophisticated technology.
You don't necessarily need to use a publicly available cryptocurrency that exists in the wild today. If an organization was motivated enough and thought it was a good idea, they could simply launch their own private blockchain on their own hardware that they control. With your own blockchain on your own hardware you could still reap the benefits of decentralized compute power, and build your own smart contracts system that can broker chargebacks. The barrier to entry for an endeavor like this is the lowest its ever been too, you can even spin up a managed blockchain on AWS these days.
https://aws.amazon.com/managed-blockchain/