At an old job, my boss asked me to take a look at a business critical application to see if it could be improved upon. It had some deficiencies that were really hampering things.
I got the source from a coworker (this is when I worked on rockets, so none of us were or are professional software people, but even my dumb engineer self knows this is not how it should be). I opened up the folder. Lots and lots of files with little apparent structure. Ah, there was a file called "MAIN". I opened it. Visual Basic 6. Over 29,000 lines for global variable declarations alone. The actual program logic heavily used a commercial library designed to (shudder) make VB6 development more like making macros in spreadsheets. The original programmer then implemented the main program logic in this hellish abomination. The logic took up another 50,000 lines or so.
I told the boss that it would be easier to write a new program from scratch then attempt to understand it.
Has anyone seen a program with more globals than this?
I only lasted there 6 months before I left for a start up, in many ways it was too much to bear. A while after I left taxation for retail changed substantially, there was a massive IT stuff up and the billion dollar business wound up being carved up and sold off; several thousand jobs affected, many lost their jobs either directly or indirectly as a result.
Sure this is probably at the worst end of the scale (short of causing deaths) and is rare but it does happen. Billion dollar company succumbs largely because of IT failures. It got to the point that the cost of the technical debt was higher than the carve up losses...
In Cobol 85, all variable are global to the program, and statically allocated at program start up. It's not uncommon for a Cobol program to import 10's or 100's of 'copybooks' and have 100-1000's variables in the global scope. They are, however, name spaced after a fashion.
My experience with this came from a similar behemoth mainframe app; 35 MLoc, 15,000 programs, thousands of jobs, running a batch and interactive green screen app that a company hosted. Was the source of about $500M a year in revenue.
Generated code? What do you mean? (I have no familiarity with VB6 at all. I have written quite a bit of code in C, Python, and MATLAB). Besides, all this was handwritten (I eventually had a conversation with the code's original author while at a party). The guy was crazy enough to write all of that by hand. I genuinely think the effort made him a little unhinged.
And of course he could have been lying...
I think the parent is referring to generated COBOL code.
https://github.com/ewfelten/Tracking-Report-Card/blob/master...
I'll let the code speak for itself.
What you're looking at is a horror beyond comprehension. There's much better, more maintainable ways to write automated tests.
> horror beyond comprehension
"Horror"? No, but I guess that's debateable (apparently). But "beyond comprehension"? Absolutely no. Even if we say the repetition offends our aesthetics, I can't even imagine what sort of issue you had to deal with where this file posed a problem. Anything you could possibly want to do with it should be serviceable inside of 5 minutes, except in the dumbest of Notepad-quality (i.e., little more than a text control) editors. That includes rewriting the whole thing to be less offensive, if that's your slant.
This file is nothing compared the types of things mentioned upthread, things which, again, exist in a context of actual production code.
Is it something specific to software vs. traditional engineering in Japanese culture, or is it something inherent to software in general? Given the parent's complaints about workplace culture in general, shouldn't the entire car be badly designed?
I'd say one problem is the testing process is not robust enough for software/firmware as it is for mechanicals. The tests revolve around driving the car, and variations on driving, and have evolved over years and decades. Heavy software control is newer, and in my experience was often not fully understood. Chief Engineers tend to come from mechanical. And there are always hundreds of problems to solve and limited time, and the black-box nature of firmware makes it hard to develop tests for.
A development review meeting tends include various mechanical departments, like Drivetrain or Engine or Chassis, many times they bring physical prototypes to the meeting and the problems and proposed solutions are comprehensible. The build testing procedures subject the car/parts to various mechanical tests, and long-lead tests are performed which stress parts till breaking. It's something you develop experience with an can get your head around.
Firmware bugs are not like that -- having just gone through this with my current consumer electronics startup I experienced the black box component first hand. I remember one software based problem we reviewed at Toyota, where something happened like: when the car has been sitting for >3 hours at <50F weather, and the ignition is started when the car is at a 15 degree angle (side of road) and driven on to a flat plane within 15 seconds, the "check engine" light turns on. Even though we had no test for that, we heard about it for customer complaints came through. But the fix is to have the supplier (in this case, but could be the Toyota engineer) fix that problem, with most of the discussion centered on timing and certainty of fix. I don't remember any discussions about global variables or lines of code.
promotion is based on age and length of employment. employment is essentially for life. you can't really be fired and you if you quit you may never again be hired (and i can't imagine whistle blowing would bring you any better). seniority reins supreme and unquestioned. there is a huge focus on hard work and long hours but also on loyalty and face.
contrast this to where and how you're working and think about the differences it would make. my friends were constantly frustrated that, try as they may, they could not elicit feed/push-back from anyone that worked under them. part of this is them being stupid foreigners that can't "read the air" properly, but part of that is people being extremely hesitant to touch anything that could be considered a break in seniority or disloyal. and this was a start-up. these were the f'in cowboys in comparison.
from what i gather, this system worked out pretty good with the long, patient and deliberate development cycles in hardware. when a product cycle takes ten years it might not be that bad to have people running it 'cause they've been there 10-20 years. but it doesn't work very well in software.
How it is a hardware-first country. Many reasons were given -- cultural, how "real men build real things", infatuation with appliances, which are seen as being "hardware" in a way, perhaps the langauge barrier i.e. existing software tools using English for menus and inputs etc.
And it is kind of interesting, looking back at a country that many in the 80's feared with surpass and leave the Western world in the dust technology-wise, we haven't seen that much software produced there. Can you think of large well known software from Japan (excluding console games). Vine Linux, SoftEther VPN, TrendMicro, anything else?
One wonders if they see Western software companies and if there is any desire to catch up or try to keep up.
Maybe anyone from Japan can provide a better perspective?
They occasionally send some of their engineers to our offices (for 6-12 months usually) to learn about our culture. There is a clear disconnect when they send these people. They work hard, no problem with long hours, but they have a very hard time getting into the collaborative nature of our teams, and I don't think it is just the language barrier. I guess I take it for granted usually, but it made me realize how much time we spend discussing things between developers, questioning management, writing furiously on the whiteboard, arguing, trying things out, failing, trying something different, finding problems with other people's code, fixing said problems, etc. It can take a bit of thick skin to absorb such criticism and engage in this process, but I think it is crucial to good software development. Someone who just sits quietly in meetings, then waits for a task to be assigned, and works at that task until it is complete is somewhat missing the process, because sometimes it turns out the task your manager assigned doesn't make any sense, or can't be done with this framework, or that you would have to use 100 global variables to accomplish the task this way.
I am merely speculating but it's possible that a very rigid social and management hierarchy where it is more important to work hard and respect your position in the company than to solve the problem the right way if it involves bucking authority, could negatively impact the ability of developers to innovate and/or produce a quality product.
Ruby.
Aren't car ECUs "appliances" without API access, just like games?
Which is very interesting, because in US it is completely opposite (and frankly it is not good either). Is it because US deindustrialized itself last 40 years?
As hardware limits were relaxed over the years, what you could accomplish in software, even with cheap hardware, grew so enormously that the primary value was increasingly in the software. This was such a rapid transition in the computer industry that it nearly destroyed IBM, it propelled Microsoft to power, and I ended up with a career in software.
Toyota and other Asian "hardware" vendors are in some sense where IBM was. They sell prize-winning hardware and whatever software that comes with it is essentially free. (Apple is a lot like this, which is why their hardware keeps getting better and their software--well, did I mention the hardware is thinner?)
The Toyota "platform" still can't do much in software, but it's growing exponentially. Eventually the software in a car might matter more than the (commodity?) hardware, but that's not the world in which Toyota managers were formed. They are obsessive about hardware, but software is an afterthought. It's for guys who couldn't cut it as "real engineers", as was the case in the American "computer industry" until the 1980s.
I also lived and worked in Japan. I was a strategy consultant, and I was told by some big Japanese and Korean companies (whom everyone has heard of) to emphasize my hardware background and not be seen spending too much time hanging out with the software guys if I valued my reputation.
?
That doesn't sound applicable to the USA, which currently produces more goods than ever before, and more than any other nation except China.
And sometimes, quitting is the ethical thing to do.
Handing your concerns to management in writing isn't really a done thing in this culture. I sometimes get laughed at for even suggesting it. And people who quit often take a massive pay cut, are forced into contract work with no safety, or even don't ever get hired in software again - especially in Nagoya where everyone seems to knows everyone.
And when you have a family and a mortgage to pay (selling a house here is also a guaranteed massive loss), it can realistically be a choice between you and your family on the street or doing what you're told.
While the engineers aren't blameless, the system over here is really broken.
people who quit often take a massive pay cut,
are forced into contract work with no safety
Is this due to the salaryman culture in Japan? I'd probably have a heart attack if I had to work under that kind of work culture.As much as I would like to agree with you, people have different value systems. And money talks. So quitting isn't always an option, even though it is the most ethical. Can't pay the bills with ethics.
If you think quitting a dangerous project (to be replaced by another dime-a-dozen engineer by a large corporation which employs thousands) is the ethically correct choice, your moral compass is broken.
We're talking about a device sold in the millions of units that is capable of dealing great bodily and structural harm to us and our world around us.
Whistle blowing (preferably before real world harm) is the only ethical choice for someone that is made aware of such a situation.
If you're working as an engineer for Toyota's firmware it's unlikely that you won't be able to find sufficiently well paying work to keep the kids in clothes and food and keep a roof over your head.
If Toyota is doing this, it would explain a lot.
Naysayers will point out that the engine wouldn't rev up if I'm putting on the brake. That seems to be part of the symptoms of this bug. Oh, and I didn't accidentally push the accelerator instead of the brake, otherwise I wouldn't have just "bumped" the car in front.
Edit: I took the vehicle into the dealership and they said "user error".... whatever, no consumer protection in Thailand anyway, so I just moved on.
Turns out they spent 18 months writing unmaintainable code that barely worked, and the codebase had to be totally scrapped with product version 2.0. The code was chock full of cut and past, globals, and hacked mutexes- they did not even use the mutexes built into the RTOS.
Perhaps Denso stuck their B-team on the project, who knows.
These two possibilities aren't mutually exclusive.
Software engineering practices aside, was it ever demonstrated that unintended acceleration could actually arise from the behavior of this software?
That's entirely the wrong question to ask. The correct question is "has the system design been proven to make it impossible for unintended acceleration to occur?"
No, but the software engineering practices were so bad that it was a moot point.
Software engineering is apparently the opposite.
You. You were one of the many people I had no clue where they came from. When I first saw the reports they seemed legitimate and Toyota seemed to be hiding something but so many people just didn't believe it and the news just sort of went under the radar.
This is done to reduce stack usage; it does not mean that these 10000 variables can be accessed from everywhere else.
But I don't want to ruin the "hurr-durr, stupid C programmers" party that the rockstar full-stack webdevs here like to celebrate. Obviously, they know more about embedded software development than people who have worked in this field for several years.
So this isn't a web developer providing criticism, but someone with extensive experience in embedded software development. Perhaps reading the article linked before jumping to conclusions might be useful!
Sadly, you would be surprised at the number of embedded systems that store and pass around their state using global variables, despite the obvious stupidity of that approach.
(For the record, I am not a web dev and I did indeed spend several years working in the field of embedded development.)