Working in embedded systems, also with a control background and software affinity, I find we have the same mindset problems in non-manufacturing contexts as well (any kind of industry that needs embedded systems).

>In my opinion, this industry is as ripe for disruption as it gets- if you can generate and update code from a high lvl object and process description here, and proof reliability and speed + allow for user-code - the industry would throw money at you.

They won't, because they don't see the value in doing it right. An industry managed by mechanical engineers is not going to see how they're spending 100x as much as they need to for a worse product. Because they don't see the value in software in the first place. For them it's just this nasty thing you need to throw a lot of money at until it everything works. The mindset would need to shift to software being the main space in which problems are solved, because in 2018 it is. I guess this also turned into a </rant>

> They won't, because they don't see the value in doing it right. An industry managed by mechanical engineers is not going to see how they're spending 100x as much as they need to for a worse product. Because they don't see the value in software in the first place.

This has largely changed in the automotive industry over the last few years. We still aren't quite there yet on getting all managers to spend on security preemptively, but at least we now have had the big honchos hand down the gospel that software is important.

I hear ya. The issue, though, is that the end product has to be at least readable, if not maintainable, by any half competent electrician, so lowest common denominator (ie. usually ladder logic) is as abstract as you can go. Also for most industrial controls, the system is "wide" rather than "deep", in the sense that almost everything is pretty trivial ('read inputs X and Y, set output Z if X and not Y') but there are thousands of X and Y. And finally, X etc. are tied so intimately with the hardware, which by the time the site is actually built and all the vendor packages integrated is only loosely related to the original design. So while I absolutely agree that it could be improved on significantly, the end result will still look (on the surface) quite like what we have now.

> the end product has to be at least readable, if not maintainable, by any half competent electrician

You can get that by using Python or similar languages; you can get that by programming in something like a Haskell DSL and hiding the underlining libraries; you can get that by creating an actual DSL and interpreting it on the fly.

But C or C++ state machines manually encoded into structured code is one of the things that won't get you there.

Absolutely not. Most industrial automation engineers, let alone electricians, are unable to handle Structured Text (the standard IEC text-based programming language for PLCs), let alone Python. Haskell is not even on the same planet.

And a DSL is the exact opposite of what we're trying to accomplish: An electrician with no prior knowledge of the system needs to be able to read the code and understand which switches and sensors need to be turned on in order for that motor over there to start. Having to learn an entire new custom language at 3AM when you're losing $700,000 an hour in lost production is not an option.

(Edit: I should clarify that I'm talking about mining here. High end factory automation engineers may very well be more familiar with 'traditional' languages.)

The people I've worked with who are operating power plants can't read code. Perhaps they can read a schematic and could understand ladder logic. They are usually quite skilled mechanically though.

Even the automation professionals I work with whose jobs are to write and maintain PLC programs look at the 'structured text' PLC logic I've written (a constraint solving algorithm) and can't be bothered trying to understand it.

Controlling a physical system is different from programming computers. The simpler the logic is to read and understand the better. Direct and verbose is better than layers of indirection and abstractions.

The systems where wide, and not deep. But by now they want industry 4.0 - means they want intelligent behaving machines. As in - material flow on fail rerouting, process controlling and self-adjusting, as in self checking for upcoming flaws. All parts are tracked over network, all guides to the machine have to be electronic.

And they want this with the conditions you described.

Is this in factory automation? I work in mining and everything is still very simple there, nothing more complex than a PID controller or speed ramps on some variable speed drives. I wish we could do some more advanced tuning, but the key criteria for anything we do is reliability, robustness (even in the face of sensor failures, everything is monitored and has manual overrides) and simplicity.

OOP isn't always needed. Most of the PCLs are very simple state machines where there really is no actual use for inheritance, or even encapsulation.

It it true though that the mindset is decades behind the web-oriented world.

Then again the web oriented world all too often come across as architecture astronauts...

Its simple at the start. And sorry, but with the stuff i work, its usually not enough.

OO would allow for easier code reuse, by composing objects out of pre-existing objects, something that is usually done by copy paste today. OO would enforce encapsulation and prevent global periphery fondling by serveral free running state-charts, where its really tough to find out, who in what order did flip the switch and cause a crash in 1 of 1000 runs of machinery.

List goes on... Sorry, but for my use cases the complexity is usually not completely avoidable.

Did you ever have the pleasure of working with Siemens' SCL? I found it very frustrating to use but overall a better experience than just using the lego blocks programming you usually see. Combined with your data blocks you could at least get some structure into your programs.

I did, i did and i used the TC synonym.

The problem is- as with all state-charts fast grows in complexity to a level where nobody has a overview. Are we the first industry to encounter this. No- chip designers have this all the time, game industry has this all the time, every fucking industry using state-charts has this - all the time. So how comes, we are the only industry taking a lot of overtime all the time?

<chirpin ciccadas since decadas>

I have people whos VMs literally collapse under page-long state-charts. So how about breaking them up, just have small state-charts in FBs and small state charts in FBs marshalling them.

Not happening. And they use assembler, not for a final tweak but as base language.

And usually, when the whole mess is collapsing in on itself, like a black hole of bad design, the project manager call in some external consultants, who should be happy to work a project that is "that far along -its allmost done".

A castle made from dinosaur bollocks.

I think the big problem in us being that slow is that you can't properly test changes. The initial programs get written under immense time pressure to put it productive and what after that? Do you want to touch your productive system and refactor it and risk causing hundreds of thousands in damages for production loss? You pretty much need to nail your initial design and can't afford to do the usual "we will take care of our technical debt later".

I hear you saying "but you have debuggers/simulators!", for anyone actually having worked in the field you will know they are pretty much useless for big changes in machines that speak to hundreds of other systems, sensors, motors, etc. At least that's what my experience was and nobody has a backup factory to test changes on. In the tight schedules we had during production stops (Sunday nights, mainly) we were busy enough maintaining everything else than having fun on a PLC.

I completely agree that this is all a big mess and people don't tend to write maintainable programs, they just want their machines running and this is where everyone needs to be trained and improve.

Manual simulators don't give much benefit, manual testing in simulation is only marginally better than manual test on hardware.

Need automated tests in simulation, preferably based on data collected in real systems. Ideally also hardware in the loop testing.

> So how comes, we are the only industry taking a lot of overtime all the time?

Have you never met a game developer?

Don't know why the downvotes. I was once asked to work on the original Gears of War. After talking to the developers at Microsoft game studios, I noped on out of there. Most game devs I've met have horror stories of death marches followed by massive layoffs. It's... not a great sector of the tech industry, if you value work-life balance.

I would hope that the cycle of crunch -> ship -> layoffs is in the past now. From memory it used to be that way because the studio needed to crunch to get the product out the door to sell. Then once that happened they had no need for all of the people since there wasn't another product to work on.

These days there are a lot of studios working on so called perpetual games (MMORPG, etc) which have constant maintenance and improvements so they don't need to let go of people. Also the other AAA game companies are bigger and can shift people between projects as needed.