News for hackers. Stuff that matters.
Re:Stratus: Servers that won’t quit – The 24 year running co... by jcoffland on Sunday January 29, 2017 @02:17PM (#13515986)
I'd be more interested in the computer system with the longest uptime. I've had Debian systems that have achieved over 1 year uptimes. Could have gone longer but kernel upgrades took precedence over uptime.
Re:Stratus: Servers that won’t quit – The 24 year running co... by nickpsecurity on Sunday January 29, 2017 @02:27PM (#13516045)
OpenVMS or AS/400. Very common for admins to report systems running for 5+ years without crashes while doing real work. People occasionally forget where they're at and have to go looking for them. My company has an AS/400 employees never saw go down or get maintenance over 10 years. It's in use 24/7. They might have some fail-over setup or upgrade when nobody is using it due to traffic patterns. We've just never seen it fail. Their other benefit is they're very hands off with it nearly self-administering.
Re:Stratus: Servers that won’t quit – The 24 year running co... by protomyth on Sunday January 29, 2017 @03:36PM (#13516464)
Or that AS/400 calls the tech to fix it and s/he just does it while no one is around. Had a call once from IBM asking for directions since the AS/400 RAID cache battery needed replacing. No danger of losing data, the AS/400 was just running slower because it was bypassing the cache.
Re:Stratus: Servers that won’t quit – The 24 year running co... by rashkov on Sunday January 29, 2017 @03:48PM (#13516547)
So... a tech would just show up at your office, ready to fix the machine? On the one hand, that is some fantastic service when you're running mission critical software. On the other hand, that must cost a whole lot of money and is quite devious of IBM.
Re:Stratus: Servers that won’t quit – The 24 year running co... by seanp2k2 on Sunday January 29, 2017 @04:16PM (#13516713)
Yeah, that type of service and phone-home is also common with $$$ enterprise storage (SANs). Funny story: in one instance I know of first-hand, the people running a NetApp filer setup with a redundant pair of systems had chosen to disable AutoSupport (NetApp's version of the phone-home support system where a tech would just show up with replacement disks when some failed) and take on support themselves. They did not test the alerting well enough after some network changes, which left the poor SAN unable to get distress signals through.

One day, the shared mount on a ton of systems around the company just stopped working. The master node (these have internal redundancy at the network, controller, shelf, and disk level) had failed over to the redundant node, which after time had so many disks fail without being replace that it went down hard. They had to work with NetApp support (at great expense) to get it back up, which took a few days. The system had been attempting to warn the admins as best it could of the impending doom for the better part of a year, and no one noticed.

The moral of this story is that you can have big-dollar redundant systems, but if you don't also have some dollars going to the care and feeding of them, it's wasted. The other lesson here is that such systems don't exist in isolation and are only as good as their weakest link, which in this case was a now-unreachable SMTP server. Lastly, although the LCD displays were trying their best to also signal this fault condition, people who would walk by these physical machines were used to this, so they didn't bother to actually read the errors. Do that at the peril of anything dependent on that system.

Re:Stratus: Servers that won’t quit – The 24 year running co... by syntheticnature on Sunday January 29, 2017 @07:07PM (#13517663)
"The Fail-Safe Theorem: When a Fail-Safe system fails, it fails by failing to fail safe."

--https://en.wikipedia.org/wiki/Systemantics#System_failure

Re:Stratus: Servers that won’t quit – The 24 year running co... by bcbrown on Monday January 30, 2017 @05:22PM (#13525627)
I love that book. Have you read https://www.amazon.com/Introduction-General-Systems-Thinking...? It's similar, but a bit more rigorous.
Re:Stratus: Servers that won’t quit – The 24 year running co... by throwawayish on Monday January 30, 2017 @04:41AM (#13519720)
> The system had been attempting to warn the admins as best it could of the impending doom for the better part of a year, and no one noticed.

This is the area were inverted Unix philosophy applies: Give me good news as well. A weekly mail, that backups (or the filer or RAID or whatever) works A-OK. If the mail stops coming, something broke.

Re:Stratus: Servers that won’t quit – The 24 year running co... by hackuser on Sunday January 29, 2017 @04:45PM (#13516893)
> The moral of this story is that you can have big-dollar redundant systems, but if you don't also have some dollars going to the care and feeding of them, it's wasted.

Agreed. Another moral: Don't believe anything works until you see it with your own eyes.

Re:Stratus: Servers that won’t quit – The 24 year running co... by protomyth on Sunday January 29, 2017 @07:24PM (#13517723)
Had a friend unplug the UPS to test it while I was still configuring the new machine. "Can't really be sure unless you test it"
Re:Stratus: Servers that won’t quit – The 24 year running co... by throwawayish on Monday January 30, 2017 @04:43AM (#13519725)
Especially true for backups. Make sure those aren't write-only!
Re:Stratus: Servers that won’t quit – The 24 year running co... by shubb on Sunday January 29, 2017 @06:30PM (#13517492)
Note to self, if need to socially engineer my way into the equipment room at a large company, pretend to be an IBM tech answering a phone home...
Re:Stratus: Servers that won’t quit – The 24 year running co... by protomyth on Sunday January 29, 2017 @07:21PM (#13517705)
Good luck with that, as we do check these things before we allow folks to get to the servers. IBM has been doing this for a fair bit and we aren't exactly blind idiots.

It would be interesting where I am since you skip local crime and go straight to federal offense. Kept me away from booze in high school since its not minor in possessions, its federal bootlegging. I can only imagine what trying to enter a server room nets a person.

Re:Stratus: Servers that won’t quit – The 24 year running co... by teh_klev on Sunday January 29, 2017 @05:41PM (#13517218)
We just decommissioned a pair of EMC CX's. These things had phone home to EMC (and then eventually a third party maintainer) if something was about to go bad or had failed. We'd usually have an engineer onsite within four hours at the DC with all the right parts to carry out a fix.

By the time it was being looked after by the third party maintainer they'd just leave us a few disks and a couple of fibre channel controllers onsite and we'd get the DC remote hands to swap them out. The faulty parts were then boxed up and couriered back to the maintenance company all on their dime.

There was not an awful lot of involvement on our end except for authorising access to the DC when an engineer needed to attend.

None of this was massively expensive (and this was 24x7x365 cover) in the great scheme of things given their mission critical status.

Re:Stratus: Servers that won’t quit – The 24 year running co... by jonknee on Sunday January 29, 2017 @04:07PM (#13516660)
> that must cost a whole lot of money and is quite devious of IBM.

You have essentially described IBM's business model. You pay more, but if something goes wrong someone shows up.

Re:Stratus: Servers that won’t quit – The 24 year running co... by protomyth on Sunday January 29, 2017 @05:10PM (#13517051)
It actually isn't that bad on the budget. We don't have the same day service (I think its 24 - 48 hour response time). It sure beats the heck out of waiting for the disaster as the thing is pretty proactive. Much, much cheaper than service and support from Oracle, and Oracle doesn't visit or monitor the machine.
Re:Stratus: Servers that won’t quit – The 24 year running co... by mavrc on Sunday January 29, 2017 @11:49PM (#13518770)
> So... a tech would just show up at your office, ready to fix the machine?

Yes. They are expensive and deviously good at it. Or, at least they used to be many years ago.

Many years back I worked in a tiny town (1000 people?) four hours from the closest depot and we even got same day service. This cost a king's ransom. But the AS/400 ran most of the business...

Re:Stratus: Servers that won’t quit – The 24 year running co... by nickpsecurity on Sunday January 29, 2017 @04:15PM (#13516710)
It's possible. I do want to try to get the uptime info from the people that run it if I ever see them. We regularly get techs come in to improve or fix IT systems. Just not that one.
Re:Stratus: Servers that won’t quit – The 24 year running co... by protomyth on Sunday January 29, 2017 @05:14PM (#13517071)
I have had multiple years of uptime until I needed to do an OS upgrade for some new part of the financial system.
Re:Stratus: Servers that won’t quit – The 24 year running co... by nickpsecurity on Sunday January 29, 2017 @07:28PM (#13517737)
That's what vast majority of my feedback has said. Stays up until OS upgrade. Whereas, my research on VMS side indicated they do clusters that support rolling upgrades of the individual machines to not affect application. Might be around that remaining limitation of AS/400 uptime if they have software for that. Do they? Or are customers forced to upgrade to mainframe with Sysplex?
Re:Stratus: Servers that won’t quit – The 24 year running co... by protomyth on Sunday January 29, 2017 @07:55PM (#13517862)
OpenVMS has an amazing cluster system, and yes they did upgrade one machine at a time in the cluster and lose no overall uptime.

I think the bigger IBM z machines do allow zero-time upgrades, but the iSeries (new name for AS/400) is basically a reboot the machine affair. I've never heard of Sysplex in regards to the iSeries.

They sent me two DVDs with 12 pages of instructions. The unnumbered first page is release notes and a notice that physical media deliveries would be an average of 5 to 9 business days on Jan 31, 2007 and beyond. (odd notice for a 2016 document)

Page 1 has my customer information with all sorts of tracking numbers. It also include the number of selected files (1,360), Kilo-bytes (sp) of data (3,963,529), fixes (3,394 fixes with 815 Kilo-bytes of data were superceded), etc.

Pre-install instructions finally show up on page 3 and the big warning about f'n it up shows on page 4. Page 6 brings us to the actual install which is written to be typed exactly with exact detailed results of each step.

to give an example

  2. Load the image catalog into the virtual optical 
     device using the following command:

  LODIMGCLG IMGCLG(ptfcatalog) DEV(OPTVRTxx) OPTION(*LOAD)

  3. Type GO PTF and press the Enter key

  4. Take menu option 8 and press Enter key
Took about an two hours counting the the 45 minute reboot. We have a very old and slow machine (need to buy a new one in the next two years, we are at a decade of use now) so reboots are a bit slow.
Re:Stratus: Servers that won’t quit – The 24 year running co... by skissane on Sunday January 29, 2017 @04:38PM (#13516856)
Do AS/400 and OpenVMS systems support live kernel patching? Can you patch the entire OS (even the lowest layers) without any reboot? If they don't, then you can't have zero downtime for years unless patches aren't being applied, which is rather bad for security.
Re:Stratus: Servers that won’t quit – The 24 year running co... by gaius on Monday January 30, 2017 @03:43AM (#13519504)
With VMS you can hot add a machine to the cluster, migrate all running processes to it, then shut down the first node without any interruption. People kept systems running while moving buildings like this. And this was 20+ years ago.
Re:Stratus: Servers that won’t quit – The 24 year running co... by youdontknowtho on Monday January 30, 2017 @06:50AM (#13520366)
Also supports cluster nodes with different CPU architectures. It was/is really slick.
Re:Stratus: Servers that won’t quit – The 24 year running co... by nickpsecurity on Monday January 30, 2017 @12:45PM (#13523302)
That's something people keep forgetting that should be in any "Required Features for Good, Clustering Software" document. That they could move the software from VAX to Alpha to Itanium within same cluster was pretty amazing. Also a requirement for high-availability if a processor goes EOL. The other strategy is to bet on the market leader. Something they missed. ;)
Re:Stratus: Servers that won’t quit – The 24 year running co... by youdontknowtho on Monday January 30, 2017 @02:43PM (#13524337)
Yeah, they (the new owner) is finally porting OpenVMS to x64. Their roadmap shows that this is going to be a long road, though. I can't remember their name...the company that bought VMS from HP. HP is licensing it from them for their own products.
Re:Stratus: Servers that won’t quit – The 24 year running co... by nickpsecurity on Monday January 30, 2017 @05:14PM (#13525555)
It's this company:

https://www.vmssoftware.com/about_faq.html#faq1

They didn't buy it. HP is too greedy about the lucrative revenue & maybe patents they get off VMS that made me wonder why they killed it in the first place. Instead, it appears (not certain) that VMS Software Inc gets some OEM-like license that puts them in control of it while HP still owns it, can license it themselves (probably existing customers), and likely gets a portion of VMS Software Inc's licensing revenue. So, they've pushed off almost all responsibility for it onto another company without actually selling it or losing all the revenue.

Still a good development for VMS customers. Both legacy and people who will suffer through archaic stuff to reap benefits of bulletproof clusters. I'd be happy to. :) Only problem is it's basically got no security attention over the years with plenty of zero days or configuration issues lurking in there. If I deployed in a company, I plan to put network guards in front of it to absorb attacks while converting the traffic into something easy to process. Ideally, a PCI-based guard that forces it to only talk to the application's memory instead of rest of system. Plus safe-coded apps (Rust, Ada) and API wrappers to try to spot BS. That VMS supports cross-language development will probably help.

EDIT: The weird thing is they finished a port to Alpha ISA before they got to x86 ISA for migrations. I thought Alpha's already migrated to Itanium. Didn't see that coming lol.

Re:Stratus: Servers that won’t quit – The 24 year running co... by nickpsecurity on Monday January 30, 2017 @12:47PM (#13523322)
Here's a guide on the main techniques it used for high-availability:

http://h41379.www4.hpe.com/openvms/whitepapers/high_avail.ht...

Re:Stratus: Servers that won’t quit – The 24 year running co... by krylon on Monday January 30, 2017 @09:30AM (#13521420)
There remains the subtle distinction between downtime and unplanned downtime...
Re:Stratus: Servers that won’t quit – The 24 year running co... by bluedino on Sunday January 29, 2017 @05:43PM (#13517234)
Most people recommend an IPL of an iSeries (AS/400) at least every 1-3 months. Did they just do that after hours?
Re:Stratus: Servers that won’t quit – The 24 year running co... by srssays on Sunday January 29, 2017 @02:40PM (#13516132)
Re:Stratus: Servers that won’t quit – The 24 year running co... by donatj on Sunday January 29, 2017 @04:36PM (#13516843)
I had a Win2K AS file server that would go years at a time after MS stopped pushing security updates. I switched to a Linux micro server a while ago and now currently use a Mac mini. The Mac Mini honestly has had the worst uptimes of the three.
Re:Stratus: Servers that won’t quit – The 24 year running co... by aidos on Sunday January 29, 2017 @02:52PM (#13516196)
I was asked to look at something on a Debian box the other day and logged in to discover over 2 years have passed since a reboot. A bit dangerous, but also wonderful to think about all that data flowing through without issue.
Re:Stratus: Servers that won’t quit – The 24 year running co... by matt_wulfeck on Sunday January 29, 2017 @03:55PM (#13516582)
I love the idea of that, but it also means no security patches for that same period of time. I'd be weary of placing that host in any network with publicly-routable hosts.
Re:Stratus: Servers that won’t quit – The 24 year running co... by delinka on Sunday January 29, 2017 @04:26PM (#13516789)
No security patches to the kernel. You can certainly upgrade user-space items (daemons and the like...)
Re:Stratus: Servers that won’t quit – The 24 year running co... by syrrim on Sunday January 29, 2017 @08:30PM (#13517995)
I heard you could hot patch the kernel as well somehow.
Re:Stratus: Servers that won’t quit – The 24 year running co... by PeCaN on Sunday January 29, 2017 @08:37PM (#13518032)
You can use stuff like https://github.com/dynup/kpatch or http://www.ksplice.com/ to apply security patches to a running Linux kernel, but that's still pretty new stuff.

Stratus has been doing that since the 90s, though :-)

Re:Stratus: Servers that won’t quit – The 24 year running co... by praseodym on Sunday January 29, 2017 @04:31PM (#13516817)
I've installed a dedicated webserver back in 2006 when VM's and VPS'es were less common. In 2007 it was upgraded from Debian Woody to Debian Etch and it kept doing its job until it was decommissioned with nearly 8 years of uptime. I find it quite remarkable that there we no datacenter power outages or any hardware problems (machine had a single HDD) during that period. And apparently the software (a Drupal website) didn't have any obvious vulnerabilities either.
Re:Stratus: Servers that won’t quit – The 24 year running co... by discordianfish on Sunday January 29, 2017 @02:33PM (#13516088)
At my first job, we had a sun box running that was booted once and never rebooted until being decomissioned around 5 years later.
Re:Stratus: Servers that won’t quit – The 24 year running co... by nickjj on Sunday January 29, 2017 @03:13PM (#13516305)
I had 275 days of up time on a Windows 7 box when I had updates disabled. Definitely not breaking any records, but it was a stable box.

It was very common to have many months of up time, despite a lot going on (virtual machines, games, etc.).