News for hackers. Stuff that matters.
Re:Legacy Train Control System Stabilisation (2012) [pdf] by abraae on Tuesday November 09, 2021 @12:36AM (#29158110)
> Due to the core software limitation (no source code available) we were compelled to integrate some original PDP-11 computer cards into the final product and this resulted in a hybrid PC platform.

Amazing. I've never been anywhere that lost their source code. Would love to know how it happens.

Re:Legacy Train Control System Stabilisation (2012) [pdf] by sverhagen on Tuesday November 09, 2021 @12:45AM (#29158169)
I have been in two different situations many years ago, neither of which I take responsibility for, ha-ha:

1. Contractor relationship gone sour, turned out they had withheld source code from us. This lead to a dispute regarding the code ownership. Meanwhile we were running the corresponding binaries in the wild. We ended up reverse engineering the missing parts.

2. Source code is centrally managed on a server, hard disk failure, restore not working or backups wholly missing, don't remember. This was the pre-git era, probably Visual SourceSafe, which was entirely centralized.

For me personally, let's just say I feasted my eyes and learned a lot. But I wager that there is still a lot of companies vulnerable in very similar ways today.

Re:Legacy Train Control System Stabilisation (2012) [pdf] by lagerstedt on Tuesday November 09, 2021 @03:11AM (#29159011)
Sourcesafe is a 90-era product. PDP-11 was discontinued long before Sourcesafe was even released.

There were version control software around in the 70’s but I do not think that the adoption rate was very high. I think it’s reasonable to assume that they didn’t use it in this case.

Re:Legacy Train Control System Stabilisation (2012) [pdf] by eesmith on Tuesday November 09, 2021 @03:33AM (#29159115)
The SCCS paper (https://ieeexplore.ieee.org/document/6312866 ) says the adoption rate at Bell Labs was high.

> The Programmer's Workbench has proven to be very popular with both management and programmers, and is now used by almost all software projects at the author's installation.

The followup paper https://dl.acm.org/doi/10.1145/800283.811111 at https://dl.acm.org/doi/pdf/10.1145/800283.811111 comments:

> SCCS is used extensively within Bell Laboratories. For example, at one installation more than 3 million lines of source code are controlled with sccs. These 3 million lines represent 5000 files and the work of approximately 500 programmers. Additionally, sccs is used at many installations outside the Bell System.

Re:Legacy Train Control System Stabilisation (2012) [pdf] by skissane on Tuesday November 09, 2021 @04:07AM (#29159318)
> The SCCS paper (https://ieeexplore.ieee.org/document/6312866 ) says the adoption rate at Bell Labs was high.

So, SCCS runs under Unix. The software package we are talking about here is Ericsson JZA715, written mostly in Pascal with a bit of PDP-11 assembler, running under RSX-11M, it was first put into operation in Oslo in 1979. I don't know what development platform Ericsson was using, but I doubt it was Unix, seems far more likely to have been RSX-11. What version control systems would Ericsson have been using on RSX-11 in the 1970s? I doubt it was SCCS – could it have been something else? Anything? RSX-11 has a versioned file system, could they have just used that? Or did people build more advanced version control systems layered on top of that base?

Re:Legacy Train Control System Stabilisation (2012) [pdf] by bregma on Tuesday November 09, 2021 @06:50AM (#29160421)
I played with SCCS on RSX-11M in the 1980s. SCCS is not a client-server architecture that allows you to share file history across the internet. It allows you to keep change history locally with your local source files. If you don't keep viable backups of your local files offline and offsite, it's going to be easy to lose even if your local files contain the entire history of your changes. It was only when we moved to DOS that SCCS became useful, since you could back up and share code using floppies.

When I worked on RSX-11M, and later VMS, we did not rely on SCCS because it had no added value. We relied on the file versioning provided by the file system and rarely kept more than one old version around because disk space was always at a premium. Changes to production code required filling out a paper form and passing it to the test department, who would submit the appropriate batch job to rebuild everything, then copy your changed files into the production directories after the tests pass. It was just like modern CI systems, except for the fact that it was mostly manual and a build-and-test cycle was at least 2 days.

Re:Legacy Train Control System Stabilisation (2012) [pdf] by eesmith on Tuesday November 09, 2021 @04:42AM (#29159543)
Quoting the 1975 paper: "There are two implementations of SCCS: one for the IBM 370 under the OS and one for the PDP 11 under UNIX."

The original version was for OS/MVT and implemented in SNOBOL4 using the SPITBOL compiler. https://apps.dtic.mil/sti/pdfs/ADA084326.pdf says that SNOBOL4 was available for the RSX-11D by 1979.

I have no clue what Ericcson was using. My inference from the literature is that apparently when software developers have access to a version control system, they used it, even back in the 1970s.

FWIW, I regard versioned file systems as a form of version control.

Re:Legacy Train Control System Stabilisation (2012) [pdf] by skissane on Tuesday November 09, 2021 @05:06AM (#29159695)
I wonder if the source code or binaries of the OS/360 version of SCCS are extant.
Re:Legacy Train Control System Stabilisation (2012) [pdf] by eesmith on Tuesday November 09, 2021 @06:50AM (#29160418)
I checked archive.org but didn't find any. I assume there are collections of IBM 370 software but don't know where to look.

I did come across https://books.google.com/books?id=Gf4YAQAAIAAJ&pg=PA89&dq=%2... from 1978 ("Software Designers Workbench", Paul A. Scheffer, Martin Marietta Aerospace

> One of the more innovative of recent software engineering activities is the concept of a Programmer's Workbench (PWB)

(PWB is the term for the product which included SCCS.)

> A simple generalization of the PWB concept results in the idea of a total software engineering facility, i.e., a Designer's Workbench (DWV) ...

> Using a PDP 11/70 architecture base, and system software which includes RSX-11M and the UNIX operating systems, standard PWB tools, and INGRES data base capabilities ...

Since it looks like UNIX supported the RSX-11M in 1978, why couldn't Ericsson be using Unix on an RSX-11M at the same time in Norway?

Re:Legacy Train Control System Stabilisation (2012) [pdf] by skissane on Tuesday November 09, 2021 @05:32PM (#29168433)
I asked Marc Rochkind, he doesn't have a copy of the OS/360 SCCS version any more. He said maybe it survives buried deep in some Bell Labs corporate archive, and made reference to the scene at the end of Raiders of the Lost Ark
Re:Legacy Train Control System Stabilisation (2012) [pdf] by MaulingMonkey on Tuesday November 09, 2021 @01:39AM (#29158474)
A C++ project I inherited had some binaries checked in for rarely-updated utilities - no sense in forcing everyone to build everything from scratch every time, right? Unfortunately, this wasn't done by CI, but by hand - so while those binaries could be built on the original maintainer's machine, some stale & missing source files (which said maintainer forgot to `p4 add`, `p4 reconcile`, or perhaps submit?) meant the only buildable copy of source was on their workstation, without anybody noticing.

...which was then presumably lost/reformatted some time after said dev left (quit? laid off? fired?)

Unable to build said source code, a subsequent developer added a python script to postprocess the output of said binary for their own needs. A hack, but it worked.

Then I got my hands on it. Well, I needed data that was discarded by the binary, so hacking on the python script was a non-starter. Instead, I reverse engineered what the binary was doing by stubbing out enough of the source code to get it building - then compared hex dumps of the outputs and tweaked said source code until my modified source code gave the same output as the checked-in binaries. Then I got rid of the python script by moving the logic into the binary's source code, where it really belonged. Then I made my own changes.

Didn't bother setting up CI though. Was porting a project in a heavily compressed (<1 month!) timeframe, and nobody else needed to touch said binary, and the codebase wasn't going to be used for any future projects (it was pretty terrible, and much time had been spent creating a much better codebase for future projects.) I at least made sure I checked everything in, though!

Re:Legacy Train Control System Stabilisation (2012) [pdf] by yitchelle on Tuesday November 09, 2021 @01:50AM (#29158537)
>> the codebase wasn't going to be used for any future projects

I have heard this many times, especially for the "single use executable for a specific use case". Many years later, it had made its way into production code and is on its 10th iteration.

Re:Legacy Train Control System Stabilisation (2012) [pdf] by skissane on Tuesday November 09, 2021 @03:25AM (#29159082)
The software package they were using was an Ericsson product, called "JZA 715", for PDP-11s. Did Ericsson ever give them the source code? Possibly not.

Does Ericsson still have the source code for a PDP-11 software package they sold in the 1970s? Maybe they do, maybe they don't.

Did they ever even ask Ericsson for the source? Possibly, they may have thought that modifying PDP-11 software was too hard–people with skills in doing it are hard to find these days, it is unlikely to be written in C, probably either assembly or some obscure language few have heard of–so even if Ericsson still had the source and was happy to hand it over (free of charge or for a reasonable fee), they might not have thought it worth their while to acquire it, and hence never have asked Ericsson for it.

EDIT: Actually, turns out JZA715 is mostly written in Pascal (so not that obscure a language after all) with some modules in PDP-11 assembly – see page 18 of http://srsv.org.au/wp-content/uploads/2017/02/S-38-1-Jan.pdf – the software ran under RSX-11M

Re:Legacy Train Control System Stabilisation (2012) [pdf] by h0l0cube on Tuesday November 09, 2021 @12:49AM (#29158188)
Surely they could use a disassembler and from that (and monitoring the i/o of a live system) infer a test suite from which they could rebuild the API. The new system could be tested to have the same outputs of the live system, and eventually swapped in, fixing bugs and making it more reliable in the process. But I guess that was more expensive than the FPGA emulator
Re:Legacy Train Control System Stabilisation (2012) [pdf] by ajxs on Tuesday November 09, 2021 @01:08AM (#29158295)
Reverse-engineering of even simple binaries tends to be difficult enough, without the stakes being as high as the program safely running the train control system of a major city. I can't pretend to know what conditions this was originally engineered under, but these kind of systems today are subject to intensely rigorous testing, and sometimes formal analysis. In a safety-critical system it's just not going to be worth doing that. Plus, if the original source code has been misplaced, the original requirement documents are probably gone too.
Re:Legacy Train Control System Stabilisation (2012) [pdf] by h0l0cube on Tuesday November 09, 2021 @01:35AM (#29158442)
I sort-of kind-of agree with this. It is certainly why most of these systems aren't worth upgrading. But reasons why it might be desirable to wear this cost is if the original system is already showing signs of bugs, which may be true if it's being operated in new conditions (e.g., interop with new systems, increasing workload), or there are safety features that could be added but not without refactoring. But, yes, if it ain't broke, surely there's no need to fix it

Edit: Also worth pointing out that formal verification was likely not a component of building the original software, and it likely has bugs it's kept since inception. Updating the API to a language more conducive to formal verification might be an option that improves safety over the legacy system

Edit: Apparently they did replace it all eventually

https://news.ycombinator.com/item?id=29158255

Re:Legacy Train Control System Stabilisation (2012) [pdf] by richwater on Tuesday November 09, 2021 @12:51AM (#29158197)
When you have government representatives that don't understand technology, it doesn't get funded.

It's no wonder everything is emulated/old/never-replaced.

Re:Legacy Train Control System Stabilisation (2012) [pdf] by h0l0cube on Tuesday November 09, 2021 @01:06AM (#29158279)
In defense of government, they need to have an audit trail and (in functioning democracies) can be held accountable when things go seriously wrong, so it means they only have a select few orgs to choose from that have passed all the certifications/due diligence. Connex (the company running the show at the time) were a private operator who has lease of the rail system for probably long enough to justify modernizing some of the systems, but maybe not enough to justify a full rewrite. Rewrites of systems with big safety implications, like civil transport, would have a different risk profile to upgrades of less consequential legacy systems

Edit: Apparently it did eventually get replaced https://news.ycombinator.com/item?id=29158255

Re:Legacy Train Control System Stabilisation (2012) [pdf] by sushsjsuauahab on Tuesday November 09, 2021 @12:42AM (#29158149)
Breakdown of communication among key stakeholders, and nobody sufficiently motivated enough to stick their neck out :)