> It affected the system that generates up-to-date schedules for trains and staff.
...boy oh boy did trying to look into what NS uses for crew scheduling ever send me down a rabbit hole.
I don't know if the systems have changed but while poking around online I found this[1] doc about the Netherlands' timetable revamp around 2006 and it talks about the complexity of TURNI-- their on-the-fly crew scheduling system.
> A typical workday at NS includes approximately 15,000 trips for drivers and 18,000 for conductors. The resulting number of duties is approximately 1,000 for drivers and 1,300 for conductors. This leads to extremely difficult crew scheduling instances. Nevertheless, because of the highly sophisticated applied algorithms, TURNI solves these cases in 24hours of computing time on a personal computer. Therefore, we can construct all crew schedules for all days of the week within just a few days.
Then I found more detail about TURNI's implementation in this[2] paper about optimizing crew scheduling for timetables.
> In the railway industry the sizes of the crew scheduling instances are, in general, a magnitude larger than in the airline industry. Moreover, crew can be relieved during the drive of a train resulting in much more trips per duty than typical in airlines. In other words, the combinatorial explosion is much higher. The latter has made the application of these models in the railway industry prohibitive until recently.
Cool stuff.
Finally, gleaning from ns.nl's careers page[3] everything else in their IT land outside this system runs off SAP (likely including the actual distribution of the output of crew scheduling) so if I had to gamble I'd say the failure happened somewhere in the integration between them.
[1] https://homepages.cwi.nl/~lex/files/Interfaces.pdf
[2] https://repub.eur.nl/pub/11701/ei200803.pdf
[3] https://werkenbijns.nl/werkgebieden/it/sap-specialist-bij-ns...
sidenote: If anyone out there is an SAP specialist ns.nl looks like a pretty great place to work: 36hr week, 5 weeks vacation, pension, and free unlimited 2nd class + low cost 1st class train travel.
Those are pretty standard terms here in .NL. 36 hours is considered fulltime by most employers, including government and semi-government. Pension is offered almost everywhere and at least 4 weeks vacation is the legal minimum.
Maybe except for the _unlimited_ train travel but most employers do offer free train travel between home and work.
This also means that if an IT system goes down on a Sunday, a lot of employees won’t even pick up the phone until Monday 9 AM.
And I'm also sure they have a significant on-call setup of the IT teams for any out of office hours issues, just like every other large 24x7 kind of organization.
It is a great place to work. A nice atmosphere, good coffee and they embraced change (in this case Scaled Agile) better than most places I have seen. They are often listed as top-5 best places to work and I can see why.
How standard is that in the private sector or non former state owned enterprises? I haven't looked at the NL market in a while but most tech jobs I saw back then were 40h/week.
Only 3% of the people in the Netherlands work 41 hours per week or more. The average Dutchman works 31 hours per week. (source https://www.cbs.nl/nl-nl/nieuws/2020/08/meer-dan-de-helft-we...)
I have a reflex repulsion for off-the-shelf "enterprise" software. You pay millions of dollars to subscribe to a suite of (seemingly) bad, bloated software that takes more work to customize and integrate than it would to fully replace. An army of developers trying to keep a business running with the world's largest Swiss Army knife.
My gut tells me that big names like SAP and Sales Force are easier to sell to executives, and it makes a company seem like it knows what it's doing. And maybe feeling pressure to choose a "respected" brand leads to fewer alternatives? Or maybe all the alternatives are quickly bought out by the behemoths. I really don't know. Maybe someone can enlighten me. It just seems like an enormous turd of inefficiency to me.
Buying an off the shelve package also buys you an industry-standard business process. The idea is that part of value comes from the process that you have to shoe-horn into. Some integration is expected (and provided for) but if you have to modify heavily, You're Not Using It Correctly.
Also, using an OTS package means less development risk (not every IT dept has 10x developers coming out the wazoo) and it also means that you have a huge reservoir of skilled labour if you need it. Not just for development but also on the business side.
Finally, if you're a big, publicly owned company, you likely have many financial / reporting constraints. The scope of your non-negotiable requirements may be bigger than you imagine.
I've seen people implement SAP in a couple of months. It's possible. It's still expensive and clunky though :-)
Never worked with SAP, but still coming off my burn-out hangover from $OLD_JOB - worked to migrate the nexus of the firm's accounting systems from COBOL to MS Dynamics. Unto itself, I don't really have a problem with Dynamics - I do have a problem with MS and their piss-poor available materials on the topic. While I understand their business sells training and certifications, this does ass all for folks pulled in as a pinch hitter during a death march. Can scarcely fly out to Las Vegas for a week long vacation while I'm working 16 hour days to beat the clock on our last true COBOL coder's retirement date.
The greater issue I have in enterprise systems - often the consultant / contractor firms[0] involved in the process. The classic fly-out of classy sales-engineers replaced with low-rent off-shore coders for the actual work. Slick-as-shit documentation that does not match reality in even the simplest of cases. Counter-Party contacts that neither care nor hide their lack of caring for the outcome of the project. Pressing for aggressive timelines that do not meet reality
I would assume there is some correlation, however, between buying $ENTERPRISE_TOOL and an unwillingness to hold consultants feet to the fire on their own failings, structuring a contract that keeps them honest.
[0] - Not to state `ArnoVW is one, only stating a pattern I've run into all too often.
In fact my line of work was managing 'complicated IT projects', generally involving formalizing things and chasing said suppliers (ie they outsourced the outsourcing)
SAP lets you stop kicking that can down the road by wasting your time and money today.
In addition, a common occurrence seems to be businesses paying for extensive custom developments (such as an integration with a third party system), and footing the entire development cost - despite their “custom” development being 100% identical to that deployed at other businesses. They’re then trapped in a world where changing the URI for an API endpoint costs €150k.
Honestly, I see it as just being a big corrupt hole fuelled by interpersonal relationships rather than any actual rationale.
That whole situation convinced me pretty thoroughly that "enterprise" software is indeed a form of large-scale snake oil. I think most businesses can go a long way with just spreadsheets and email.
You do not modify SAP to fit your company, you modify your company to fit the SAP model.
You can try it the other way, but the project will always cost millions and it will eventually fail. It always has and it always will.
I think the "Excel+" category should receive a lot more attention than it does now. I can easily see "no-code" tools like Coda taking over Salesforce and effectively filling the niche you describe.
[0]: https://coda.io
[1]: https://coda.io/@simpladocs/cycling-specific-template / https://coda.io/@simpladocs/crm-interaction-tracker
I've heard it said that SAP is sold on the golf course, not in the IT boardroom.
I have no idea how it works, but apparently it does.. sometimes…
https://www.computerweekly.com/news/252435884/Chances-of-suc...
i think this is under-emphasized for people unfamiliar with them.
I think SAP and related modules are "solidified business processes", and their consultants and sales persuade your CxO level people that switching business processes will improve overall efficiency (and somewhat hint at standardization, so that the CxO will have experience at that domain when switching jobs!).
But of course, it's not really true - best practice business processes are an illusion imho, and it will turn out that the subtle differences have to be implemented individually.
They're absolutely colossal in size
My ex-girlfiend's father is an engineer there in Baden-Wurttemberg - I'd characterise them as closer to something like Oracle rather than Salesforce
They feel more like IBM, but more crusty, more conservative and more corporate.
Working for a railway on anything touching SAP sounds like an absolute nightmare of wasting decades threading water with nothing to show for it.
(I saw a bit of SAP inside DB… nope nope nope)
Edit: no offence intended to anyone at SAP, IBM or Oracle - I do think each company has a legitimate role to play for rather different markets
Most ERPs can be quite flexible too, there's a ton of customisation possible. And a whole cottage industry of addons and integrations for every imaginable integration or business model. But the problem is you really screw yourself with added complexity when it's time to update. Then you're really condemning yourself to spending millions on consultants at a time when you might not have the resources. And postponement leads to technical debt.
Hence the idea of adapting the business to the system. Making sense too if you think about it this way: you're not really just buying a system but a methodology.
And another thing is that it makes integration with other companies easier. Makes the company more easy to sell or too acquire other companies and integrate them.
They decided to modify a number of core functions to work the same way as their old terminal system instead of the default ERP process. Now our job for the next couple years is decoupling the custom add-ons so we can safely update to a modern ERP system.
It was developed a very long time ago, it has a huge adoption in large companies and a few years ago when they migrated from R3 to Hana one of their own people told us in an architecture review meeting that they have over 300,000 built in reports to migrate.
In many cases companies are adopting SAP because "nobody was ever fired for buying IBM". Anecdotical, the company where my brother works is implementing SAP and changed all existing networking gear with Cisco because "this is what big companies do". They threw away brand new equipment and spent several millions on the SAP implementation that is still not working, forecasting they will pay even more millions to make it work then millions per year to use it.
(and bloat, and McKinsey infiltration into many a large F100+)
--
They are old school, entrenched, but yu would jump off a cliff supporting any of this legacy stuff. It will persist as the inertia on huge systems is heavy.
But yeah - I wonder if they are signing any NEW business... aside from military bunker grade stuff (which will cost as much as the bunker to implement) I dont see a reason to implement (not because I am bashing, but because I dont know the modern business case reqs that would call for SAP)
What is McKinsey infiltration?
Basically: You have a problem; Oracle/SAP says they'll solve it for you. Now you have a second problem; they say McKinsey'll solve it for you. Now you have …
God I love etymology:
"-Stasize"
**To stand on a hilltop once. To be, this time, on advance. To reach the highest clouds. To touch the limits of space. To out-grow skyscrapers. And make them look at my face.**
So FB is becoming a meta-stacious being..
That which wishes to control from a vantage, or t corrupt from within the minds... The light house.
[1]: https://blogs.sap.com/2022/03/11/what-you-may-not-know-of-pr...
people seem to believe that running trains is some simple thing. it is the exact opposite of a simple thing, even for small systems with only a few dozen trains
1 hour more than the standard French contract, and there are also far more holidays here..
> free unlimited 2nd class + low cost 1st class train travel.
.. okay that's actually a great perk. IIRC the French national railway company, SNCF, provide only a limited number of tickets per employee (like 10 per year? someone please correct me if I'm wrong). Their UX is also atrocious though (websites are just shit at workflows, apps are weird and break easily, ticket machines are OK but with really bad touch screens, and train announcement screens are each running Windows XP(with literally hundreds, if not thousands of them on big train stations), while from what i recall NS was acceptable.
There is no UX: it's just a small sheet of paper.
For example, The Salvation Army is one of the biggest jokes in all of charitable history.
I wont go into details here, but I would NEVER give a dime to the Salvation "army" (of free slave labor under the guise of "salvation" (its a tax and money fraud to its core (I went through a SHIT ton of financial documents to see what their true operatin is disguised as. AVOID)
TL;DR: TSA uses its place as a church-like service to get special tax incentives, even the buildings in SF were donated to them. They run a for-profit-grey-market of boutiques they supply with "donations" they have running trades with China-Town in SF. They take subsidies, donations and other benefits. They require all that they "help" sign up for state benefits, sign the benefits over to their org, take the benefits, then feed donated & expired processed fod to their labor force.
They only take in able bodied. Force them to go to church each sunday. and make them use external addiction help, such as AA and make it clear they are not a treatment organiztion.
They maintain a very small amount of actual employees, and play on the egos of those in their "program" using military style rankings to appease the egos of the ignorant...
They then profit like mad, corruption is rampant and they feed all their "good" donatins to their side hustle of boutiques and other grey market.
Its a fn racket.
=---
The point being that if you DD a company/org, you can often find out a lot from financial public info, comments from even just a few employees etc..
What lead you to look into this? How did you originally suspect them of being fraudulent? What clues did you see in their financial documents?
For example, I tend to look at Charity Navigator and for Salvation Army, they mostly get dinged by not having an independent board (which to be fair is a clear red flag) but not much more, so as someone who does want any contribution I make to charities to go toward their cause, I'm very interested in any methodology around researching a charity.
I knew someone who worked briefly at the Salvation Army. They said it was shockingly efficient and effective at its charitable mission.
Learned a lot about db best practices to make these systems resilient and reportable but the whole “the database is the system” angle was very much a “every problem is a nail to this hammer” situation.
And we can assume that a position in the IT department just opened up...
https://en.wikipedia.org/wiki/2017_Washington_train_derailme...
Thanks!
(Mixed metaphors always make me smile.)
The other part of my guess is that they have a SQL db which is stored on /likely/ a windows server instance. I’d even surmise that this instance may be hosted on Azure but that’s speculation, not a good guess.
On Sunday there is also not as much pressure to get things running as there would be on Monday. Or perhaps this is the incentive to pay now and get things fixed on time for the work week? In that case Friday night would again have made more sense unless the attackers have some very specific insight into how much slower restoring without paying is.
My alternative theory is an expired certificate that makes some core systems just not talk to each other anymore. The announcements on the stations, for example, were also out, and they lost control of the mobile apps (couldn't make the apps show that trains didn't run, the in-app scheduler showed all was A-OK), and that sounds quite dissimilar from the core train operation service, making me think it's more of an infrastructure than a specific system's problem. On the other hand, once the app could be controlled again (assuming this singular underlying cause), you'd think they could then also start putting train service back in place and that didn't happen for hours still.
I can't really make the pieces fit together for any theory, so then presumably something multi-faceted (one thing tripping one or two other things so it escalated from restarting one component to not being able to restart the trains anymore the whole day).
Before you can get the actual service up and running, you need to get trains and personnel to the right locations. So I don't think this discredits your theory as much as you think.
Well, as long as you're clearly stating _this_ part of your otherwise validated and highly likely pre-post-mortem is speculation... </s>
That especially. I wonder how expensive that thing is going to cost.
I would appreciate one of those post-outage "what happened" reports like with the AWS and Facebook outages last year. But outside of IT I don't think anyone really expects those over here. And there might be national security considerations preventing such disclosure until any chance of a repeat has been engineered away, at which point everyone will have forgotten.