Re:GitLab Database Incident – Live Report
by aubreykilian on Wednesday February 01, 2017 @07:06AM
(#13540055)
This happened to me one night late a few years back, with Oracle on a CentOS server. rm -rf /data/oradata/ on the wrong machine.
I managed to get the data back though, as Oracle was still running and had the files open. "lsof | grep '(deleted)'" and /proc/<ORACLEPIDHERE>/fd/* saved my life. I managed to stop all connections to the database, copy all the (deleted) files into a temp directory, stop Oracle, copy the files to their rightful place, and start up Oracle, with no data lost.
I managed to get the data back though, as Oracle was still running and had the files open. "lsof | grep '(deleted)'" and /proc/<ORACLEPIDHERE>/fd/* saved my life. I managed to stop all connections to the database, copy all the (deleted) files into a temp directory, stop Oracle, copy the files to their rightful place, and start up Oracle, with no data lost.