- Make a plan of exploratory steps that could trigger a failure, in order of increasing risks.
- Then, if necessary, get a formal sign-off from someone with authority (perhaps a director of the company) to proceed with each step.
Steps like that, which are usually very low risk but do risk taking down an unknown service, might include things like (just ideas from other comments):
- Logging in to the server at all.
- Splicing something into the network switch.
- Cloning filesystems or disk images.
- Cloning the process memory or system memory.
- Cloning the image or other things if it's in a VM.
- Running strace gdb, or packet tracing.
- Swapping the power source live.
These days, it is often possible to transfer a full working system into a VM on a more modern, powerful machine, and after doing that, it's a great relief to everyone because it's no longer hardware dependent.
I've done that with some legacy systems that were originally on bare metal, and are now still running nearly 20 years later in reliable VMs, with no change to the running software. Usually running faster and with more memory doesn't break a working system.
But doing it on a service that can't be taken down even for a moment is quite an interesting adventure! :-) It is possible, but technically challenging, as long as you can obtain disk and memory images from the live system and then capture changes fast enough to perform a hot transfer.
I’ve been in similar situations, but never without an immediately obvious solution.
That system WILL FAIL. Even as we speak the time-to-failure is shrinking.
Even if one of the solutions described below actually works, you won’t get 100% recovery.
I recall a story years ago - from MIT, if memory serves - where they rebooted a system because they had many generations of Sybase backups. When they tried, it didn’t work. Nobody had actually tested recovering from a backup.
Grit your teeth; cover your ass; and get on with it. The clock IS ticking.
I think that the key point. We have a system that can't easily be restarted and won't automatically start after a server is rebooted. The developers of the software don't care, because "what are the chances of a virtual machine spontaneously restarting". Turns out, those chances are rather good.
Servers, virtual machines, containers, doesn't matter, unless that thing is running on a mainframe, it will crash.
After sufficient CYA steps are taken, don’t offer any help on this problem unless explicitly asked to do so. And even then wait until there’s actually desperation. Feel free to think about it so you have some ideas when the time comes, but if this service isn’t something you are responsible for then stay away.
Or else it will be a “you touched it last you broke it” regardless of what actually happened. And if you make too much ruckus about it, they’ll say “why didn’t you do anything to fix it??”
When shit hits the fan if the company wanted to throw you under the bus they could delete them remotely from your machine or work mailbox and then pretend the warnings never existed.
1. Suggest that the work to replace it is prioritized commensurate with the business impact caused whilst re-establishing service as if it went down right now
2. Remind them that it will go down at the worst possible time.
3. Ensure your name is attached to these two warnings.
4. Promise yourself you wouldn't run your business this way.
5. Get on with your life.