Re:Details on the Oracle failure at JPMorgan Chase
by gaius on Saturday September 18, 2010 @04:19AM
(#1703904)
No Oracle RAC shares the same storage between two or more nodes.
What they had here would appear to be database A running on storage A which is replicated at the storage level to storage B where database B waits in an idle state. Because the replication system is "blind" - it only sees its own filesystem containing bytes, not Oracle data structures - it can't tell a good Oracle block from a bad one and copies it.
I do this sort of setup for a living and you would be amazed at how many "architects" there are around who have completely drunk the storage vendor kool-aid and don't really understand how anything works (not even storage...).
Re:Details on the Oracle failure at JPMorgan Chase
by GBond on Saturday September 18, 2010 @10:37AM
(#1704230)
This is likely the case since the post mentioned that storage controller was initially blamed (but cleared).
Re:Details on the Oracle failure at JPMorgan Chase
by jasonwatkinspdx on Monday September 20, 2010 @02:58AM
(#1708057)
I rarely work with Oracle, so I'm not very familiar with the product line. Thanks for the correction.
Re:Details on the Oracle failure at JPMorgan Chase
by georgefclay on Friday September 17, 2010 @06:12PM
(#1703014)
I did read the article that was referenced. I did not read the article that that article referenced. My point was about the comment on "over engineering". This problem was not cause by over engineering.
Re:Details on the Oracle failure at JPMorgan Chase
by chrisbolt on Friday September 17, 2010 @06:18PM
(#1703027)
It says right in the article that was referenced:
Before long, JPMorgan Chase DBAs realized that the Oracle database was corrupted in about 4 files, and the corruption was mirrored on the hot backup.
Otherwise you'd know that they had a fault that propagated to the hot spare. It's also utterly daft to think that a financial enterprise as large as JPM/Chase wouldn't already be running a HA setup. In this case it appears to be Oracle RAC.
I'm astounded how often I have to remind people that replication and backups are very different things, and that you need both.
I'm also depressed how many utterly thoughtless comments are made here on hackernews lately.