What lone developer says "let me find all the GPS coordinates of all the train repair shops and make the trains not work if they've been parked there for days"?

Don't underestimate these damned rouge developers!

I've heard of Khmer Rouge but not of Dev Rouge before.

Humor is so little appreciated these days.

Humor is prevalent in the internets various forums. It's well appreciated generally - easy upvotes on Reddit and elsewhere. But that also means that a serious forum with higher signal-to-noise ratio is something uniquely valuable.

"My cat slept on the keyboard and accidentally developed and merged working advertisement feature into a release build of the new AC game. It was totally on accident, happens all the time." (c) Ubisoft

"I come to you now with apologies. This is the third time my cat has put paid mods into Skyrim, something for which he is very very sorry you are unhappy about."

Yeah, sounds like a secret 20% project :)

Management: "a 6-pack of beer for the brilliant dev who figures out a solution to this problem of us ..."

We talk on HN a lot about ethics and programming and the importance of engineers refusing to develop unethical software.

But I think we need another PSA that, if you are going to write some code that could land you in jail, make sure you get your entire reporting chain in writing telling you to do it, up to and including the CEO and the board of directors.

Alternatively, if you're going to break the law, don't do that (break the law).

All of them. Dieselgate? Rogue dev. Trains breaking for independent shops? Rogue dev. Some airplane going down because of bad software? Rogue dev.

Repeat after me: The company is never at fault.

Perhaps GitHub Autopilot and ChatGPT teamed up to write the code upon noticing that the developer tripped and fell, causing a GSM modem to fall off their desk and into the assembly process.

Is this really suggesting some developer, buried under tasks and putting extra hours, thought to implement this feature in his free time for months, slipped it through reviews and scans and pipeline deployments, for... what? Really? I rather prefer to think the comment was sarcasm.

Yes it was sarcasm, since it is common for management to shift the blame on individual engineers in these cases (see for example the Volkswagen emissions scandal and the Boeing MAX crash).

IIRC Volkswagen actually used the term “rogue engineer” when speaking to the press.

In VW's case there was some reason to believe that at first. There are reasons to test an engine in the lab without emissions controls, and so it is believable that a lazy engineer could put in code for the lab that also triggered in real world conditions.

Note that the above is only believable before the rest of the investigation. Proper investigation proved the story wrong, but it is just possible enough to believe it could be true if the proper investigation hod come up differently.

They will surely try that explanation too if the others fail. Currently their stance seems to be "there is no such code on the trains, and if there is, it's not from us".

It’s from those darn third party developers! ;)

Blame the hacker, wait for it.

Also: I wonder if their management realizes that they probably have a nice trail in the form of a bunch of repositories and commit messages. Would be nice if that leaked.

You're talking about industrial PLCs. They're programmed using a-bit-more-fancy Scratch snappy blocks. There is no version control. The firmware contained paths embedded as strings, so we know that firmware for each model and customer was developed in a separate folder on disk. I wouldn't be surprised if they also had .zip files with backups of previous versions.

The IDE for these PLCs actually has VCS integration! It's SVN, but it's still better than nothing.

Its on-disk representation of graphical 61131-3 languages (FBD / SFC) is text-based and somewhat human readable, so there's nothing technically preventing the developers from keeping all of this in any other VCS of their choice.

There is nothing wrong with SVN, it's just that Git allows for some workflows that are better suited to larger teams and more complex projects. But for your average PLC project with a team of 10 and one binary as the output it should be more than enough.

You likely won't see any 'feature branches' or frequent merges in this kind of environment.

> There is nothing wrong with SVN

Except merging things, and handling a lot of files...

There are lots of small things wrong with SVN. But it's indeed usable.

I'm aware of repos with a few million files in it that have been going since 2003 and not a single issue.

Merging things is different than in Git but it works. I use both, and I'm not religious about either, some things are easier in Git, some are easier in SVN. Git provides more footguns. And loads and points them too.

I'm intimately familiar with PLC programming, yes, you can do it the 1980's way but there are also plenty of environments that allow for modern version control.

https://www.google.com/search?q=version+control+plc+programm...

You'd have to be pretty daft to do this kind of development today and not take advantage of version control and even the most visual versions of these systems eventually output files. You may not be able to do a line-by-line comparison but you will have a commit log with helpful messages.

Look for 'engage in anti-competitive behavior' in the log message ;)

What about: rogue hackers maliciously squashed our whole repo into a single commit?

I don't doubt that's exactly what would happen, in fact I think that that rogue hacker is about to do his thing, quick, erase the backups!

Yeah.. I've just realized that while it's entertaining to watch how it unfolds or predict what can happen next, it's also sad, because pretty much everybody in the rail industry loses..

One of my business partners works for PKP it's very annoying to see this all unfold and in this particular way. Poland has so much potential, these idiots are ruining Polands image in ways that really matter.

But then again, as a Dutch person I have enough issues locally that I can't even complain...

Ugly times.

I have a friend that runs a business, where he hires Polish developers to do his coding.

He absolutely raves about them. It sounds like he's got some good coders.

Poland has a very strong technology and mathematics tradition that goes back decades. It's one of the reasons Poland has some strong feelings about their role in the breaking of the Enigma, for the longest time that was played down.

Working in security on the operating side (albeit not in Poland):

No, pretty much just the manufacturer loses. Short term the operator loses, but I'm sure that the courts will award damages.

For me, this incident is a welcome argument with which I can tighten the screws on manufacturers in the next round of train buying (at minimum, they will agree to heavy contractual fines for anything like this; at best I get full source code for every train).

For too long the only priority in OT was safety (fine in the 80ies, but the second you integrate an IP stack that posture doesn't work anymore). This has been changing in the industry thanks to EU-regulation; this incident will accelerate the change.

What I meant is that I feel the trust among parties might go down industry-wide. In a sense you admitted that:

> (...)I can tighten the screws on manufacturers in the next round of train buying(...)

But then I can see it might help change things for the better across the board, as you nicely described. Thanks for the illuminating comment!

That's assuming we will get to the bottom of this. And I really hope we will. But I'm kind of concerned that it will all be wiped under the carpet.

Temporarily lost ok. Better to let these manufactures do whatever they want.

If this goes on to criminal charges, then they're about to discover what amazing things a thorough digital forensics analysis can find out from their workstations.

Now you're saying they're about to have a fire.

Oops, rogue hackers deleted our .zip files with the backups! :)

EDIT: BTW having no version control would be pretty telling on its own. It's a critical piece of software, that controls a train..

Do you have more information about this sort of programming? I'd like to read about it.

Google for PLC programming environment.

IEC 61131-3

ah, thank you!

If it was developed anytime after 1990 (probably before) you will find plenty of programmers willing to be expert witnesses and tell the court that the company not having version control is gross incompetence, the only reason a company would do that would be so they can hide evidence of illegal actions. As such the court should impose punitary damages.

Of course before going on the stand the expert witness will work with a lawyer to word smith the above into something the court will better understand. however I think the generic idea is something everyone here will agree with.

> There is no version control.

I've used a thing that not only doesn't play nice with versioning (your local workspace is a collection of embedded db files) but doesn't play nice with multiple developers (no way to sync workspaces). I still managed to get it into version control, even if useful things like diffs didn't do anything useful.

That didn't happen, and if it did then it wasn't us, and if it was then it was inadvertent, and if it wasn't then our current leaders didn't know, and if they did then it wasn't illegal, and if it was then we have a different interpretation of that law which is unreasonable anyway and should be ignored and repealed.

As a lone developer there is absolutely no motivation to do this and hide this decision from management.

Having said that a lone developer can come up with the idea, propose it to management with the expectation of some fat bonus. But why do this in secret?

There is just no possibility where upper management is not involved in this.

100% agree. Doing this in secret without management's express approval means no bonus, working extra for no benefit, compromising your other tasks and exposing yourself to heavy liability for compromising the country's infrastructure. This is something that companies could get away with, but a single developers would not only lose their job, but risk going to jail.

Note that management's express approval could be verbal conversations that you cannot prove happened or not. With some work management can find some way to get a developer who has some other conflict of interest to do this: if your brother in law works for the maintenance yard situations then you have motive to ensure he has work.

“Newag president Zbigniew Konieczek said that ‘no evidence was provided that our company intentionally installed the faulty software. In our opinion, the truth may be completely different—that, for example, the competition interfered with the software.’”

The ridiculousness of this defence makes it clear leadership was in the know.

I guess they can't claim any copy write / drm violations if they also deny it was their code.

Riiight?! This is a really strong defense for the white hats. Either it’s your code and you are egregious slime balls doing illegal things, or “some third party interfered and hacked your trains like you said and we fixed it you’re welcome”.

I hope evidence will be provided in court.

I'm not sure that's the smoking gun.

IIRC, there was some "konami" code to re-enable the train after disablement that was removed in a first-party update after the third-party repair company found the sequence.

And not related to the third-party locations, one of the trains had some code where if the year>2021 & month>11 & day>? then the train would disable itself; ultimately doing it in the wrong year because the train was off during november / december of that year. This is a little excessive for somebody random to do.

The 2021-11-?? date was the day the train was scheduled to be serviced (so the intention of the code was to brick the code after the service). Only by random chance it didn't work, because the train was in maintenance a bit longer, and the code didn't trigger in January 2022 because of the date checking bug.

That is completely nuts and irresponsible beyond belief.

> The ridiculousness of this defence makes it clear leadership was in the know.

https://kolejowyportal.pl/koleje-dolnoslaskie-odpowiadaja-ne... - this is from 2022-07-06. The train owner complains that they still didn't receive information from Newagg about what was fixed when unit had to be shipped to manufacturer after it refused to start.

So the issues with 3rd party servicing were publicly known well before the smoking gun was found in firmware.

Once this is proven false, they will blame the devs, just like VW did.

The dev. at VW was at least smart enough to keep the paper trail around. Lets hope the people working for this company did the same.

There's also a bunch of service people involved etc. (some "non-working" trains were sent back to NEWAG, and the people doing the maintenance had to know how to unlock them, etc.)

I wonder how they explain the "rogue dev" somehow clandestinely communicating the reset procedure to the manufacturer's shop.

It's the "you can't prove we did it" defense.

This gets penetrated by getting lower level employees to rat out their superiors.

His dog did it after eating the CS homework.

npm add block-repair-shop. It can happen so fast.

If an employee did this without authorization, the company should still be liable because I do not think safety regulations exempt manufacture from responsibility of employee actions. I believe the opposite is true, that they must ensure safety despite rogue employees, with reviews, audits and such.

They need to show audits and that the code is obscure enough that it could be missed. I've seen IOCCC type things that do things I wouldn't catch in a code review. (IIRC there is a malicious code contest where the goal is to put backdoors in that others don't see in review, but I can't recall what it is)

There's been a decent amount of those in the wild, too. If you're in an environment without unicode auditing you can do all sorts of crazy things without IOCCC levels of code tricks too.

You’re thinking of the Underhanded C contest: http://www.underhanded-c.org/. The goal is to write C code which looks completely innocent, but which actually performs some malicious or unwanted behaviour.

In the last year they ran the contest (2016), the goal was to write a program that would compare two sets of measurements for similarity (nominally radiation emission spectra) but fail to give a correct answer when given a particular crafted input.

> Oh, that seems pretty damning. I wonder if this was a lone developer or ordered from above.

Guess. (Did I miss the sarcasm again? I always miss the sarcasm.)

I suspect this is going to end all further sales of these trains outside of Poland. What railway will ever trust them again?