News for hackers. Stuff that matters.
Re:Lockheed Reassigns Workers to Fix F-35 Software by LowKarmaAccount on Thursday June 20, 2013 @02:22PM (#5913849)
> The F-35 requires more than 8 million lines of code, compared with about 2 million for the F-16 and less than 1 million for other fourth-generation fighter aircraft

The F-35 is programmed in C++, while the other aircraft are programmed in Ada, which is one of the reasons that there is so much more code. When you have to code, test, and maintain 8 million lines of C++ in an environment where a single bug can be deadly, you should expect delays.

The choice of using C++ to write a system that is designed to never crash and the obvious ignorance of the "mythical man month" does not give a good impression of Lockheed's management.

Re:Lockheed Reassigns Workers to Fix F-35 Software by jarrett on Thursday June 20, 2013 @03:49PM (#5914464)
For anyone who's unfamiliar with the term "Mythical Man Month," it's the title of a book which argues, amongst other things, that assigning more engineers to a late software project will make it even later.

The book can argue more convincingly and thoroughly than I can, but briefly: When new engineers join a project, they have to spend a lot of time familiarizing themselves with it, so they're not productive for a while. They take time away from the project's original engineers by asking a lot of questions, which actually means progress slows down.

Further, more people means more complexity to manage. There will be more miscommunication, more instances of engineers' work conflicting with other engineers' work, more differences in coding style, etc.. With more than 200 people on a project, I can only imagine the nightmares.

I'm not familiar with the F-35 codebase. But I have a hard time imagining it could benefit from 200+ people working on it simultaneously.

I'm also skeptical about this idea of 24/7 shiftwork. Will each person will have their own little piece of the codebase that nobody else touches? If so, then why have a nightshift? Why not just have everyone work in the day, since it's all in parallel anyway? If not, how in the world can you have engineer A coding a given module on the dayshift, who then hands it off to engineer B on the nightshift? Software is not like laying bricks. You can't just come on the next shift and pick up where the last person left off. You lose all the context they had in their heads while coding, which is absolutely essential.

Re:Lockheed Reassigns Workers to Fix F-35 Software by john_b on Thursday June 20, 2013 @04:38PM (#5914808)
Frequently aerospace companies will have a limited number of licenses for various pieces of software. C++ as a language obviously doesn't have this limitation, but a limited number of licenses for various custom test suites and third party software (e.g. Matlab) is frequently a bottleneck. If this is a bottleneck, it makes sense to have these licenses in use 24/7.

Purchasing more licenses is always an option, but usually not a quick one.

Re:Lockheed Reassigns Workers to Fix F-35 Software by TheCondor on Friday June 21, 2013 @09:41AM (#5917947)
Maybe it's really obvious, I mean it should be, but Lockheed clearly isn't much of a software company.

Office space and third party software licenses mandating a 24 hour work cycle for the most expensive defense project of all time?! Maybe it should not be surprising. Their peers seem to just appropriate rights and constitutional interpretation to meet their needs, can't these guys just appropriate more matlab or whatever compiler it is licenses? Just seems like it is being treated like a manufacturing problem.

This same class of folks who are making these decisions are all contracting for NSA and have access to your data to keep us all safe...

Re:Lockheed Reassigns Workers to Fix F-35 Software by jarrett on Thursday June 20, 2013 @05:04PM (#5914972)
Good point. It does make me wonder though: Does the time and money cost of purchasing more licenses outweigh the effect on employee morale resulting from night shifts?
Re:Lockheed Reassigns Workers to Fix F-35 Software by jfb on Thursday June 20, 2013 @04:41PM (#5914832)
Fascinating, thanks for the insight.
Re:Lockheed Reassigns Workers to Fix F-35 Software by outworlder on Thursday June 20, 2013 @03:59PM (#5914525)
> I'm also skeptical about this idea of 24/7 shiftwork. Will each person will have their own little piece of the codebase that nobody else touches? If so, then why have a nightshift? Why not just have everyone work in the day, since it's all in parallel anyway?

Because that way you can fit more people in the same facility? Otherwise, it will be unused for 8 hours each day (being optimistic).

Re:Lockheed Reassigns Workers to Fix F-35 Software by jarrett on Thursday June 20, 2013 @04:20PM (#5914676)
So they're running low on office space? That's a bottleneck for the F-35?
Re:Lockheed Reassigns Workers to Fix F-35 Software by jfb on Thursday June 20, 2013 @04:45PM (#5914851)
Why not? It's not like they can just go down to the local HackerNinjaRockStar DojoSpace and rent some Ikea standing desks. At Apple, for instance, there are sealed buildings and security requirements that impose a cost on expansion above espresso machines and foosball tables -- and they're not required to meet DoD legal standards.
Re:Lockheed Reassigns Workers to Fix F-35 Software by jarrett on Thursday June 20, 2013 @05:02PM (#5914959)
This is probably true. Still, it seems like a red flag to me that on a project like the F-35, which presumably has adequate funding for office space and other such things, their only solution for providing offices for their own employees is night shifts.

There's a very good reason not to have engineers work nights. While some are undoubtedly night owls, many (most?) will have lives and families which will suffer from night shifts. That accelerates burnout and increases resentment--both of which are usually running high in a troubled project anyway.

Re:Lockheed Reassigns Workers to Fix F-35 Software by jfb on Thursday June 20, 2013 @05:10PM (#5915006)
Oh, the project seems certain to be a total clusterfuck; it's just not 100% surprising that they could be gated on physical space.
Re:Lockheed Reassigns Workers to Fix F-35 Software by cobrausn on Thursday June 20, 2013 @02:49PM (#5914049)
The F-35 has multiple variants, one of which is a STOVL (Short Takeoff and Vertical Landing) aircraft with a vertical flight system. The amount of code necessary to deal with that is likely a large part of the difference in LOC, not to mention the more complicated systems for weapons, radar, communications, etc...

In other words, the fact that it is written in C++ tells you little more than the fact that it is written in C++ and not Ada, but that alone will tell you that they probably expanded their pool of potential programmers by a large margin.

Re:Lockheed Reassigns Workers to Fix F-35 Software by MichaelGG on Thursday June 20, 2013 @03:18PM (#5914241)
>expanded their pool of potential programmers

Is that true? I'd think that the main requirements would be experience, intelligence, security clearance, probably familiarity with avionics.

Any good programmer worth working on such a project should be able to switch languages.

I'd hope it's not like some corporate web site that just needs the programming equivalent of warm bodies.

Re:Lockheed Reassigns Workers to Fix F-35 Software by HeyLaughingBoy on Thursday June 20, 2013 @04:29PM (#5914743)
"Should" does not imply "wants to."

I could switch to COBOL in probably a couple weeks, but you'd have to pay me multiples of my current salary to do so for any length of time.

Re:Lockheed Reassigns Workers to Fix F-35 Software by cobrausn on Thursday June 20, 2013 @04:12PM (#5914613)
Considering the number of job postings I've seen where they want N+ years of experience, I'd say it's easier to find that in C++ than Ada if they do that.
Re:Lockheed Reassigns Workers to Fix F-35 Software by olympus on Thursday June 20, 2013 @02:37PM (#5913957)
> system that is designed to never crash

Oh, they crash. Software on currently fielded jets crash all the time, pilots call them "software anomalies."

Systems are designed to fail softly and reboot quickly (~1 minute) and sometimes do so automatically without physically crashing the jet. Sometimes the pilot won't even notice.

Re:Lockheed Reassigns Workers to Fix F-35 Software by iyulaev on Thursday June 20, 2013 @02:57PM (#5914119)
sometimes do so automatically without physically crashing the jet

I hope by "sometimes" you mean "almost every time!"

Re:Lockheed Reassigns Workers to Fix F-35 Software by olympus on Thursday June 20, 2013 @03:23PM (#5914281)
Sorry, bad grammar. I meant that sometimes they reboot automatically. The systems are designed to always reboot without physically crashing the jet, but sometimes the pilot has to command the reboot themselves.
Re:Lockheed Reassigns Workers to Fix F-35 Software by outworlder on Thursday June 20, 2013 @04:02PM (#5914534)
> Systems are designed to fail softly and reboot quickly (~1 minute) and sometimes do so automatically without physically crashing the jet. Sometimes the pilot won't even notice.

If that's the case, why not use something like Erlang?

Re:Lockheed Reassigns Workers to Fix F-35 Software by skittles on Thursday June 20, 2013 @04:41PM (#5914827)
Because you don't want to be in a high performance jet when the VM decides to do garbage collection during a critical maneuver.
Re:Lockheed Reassigns Workers to Fix F-35 Software by jfb on Thursday June 20, 2013 @02:41PM (#5913991)
I think the difference in LOC count is likely due to the much more complicated avionics in the F-35 than the technology stack, as compared with older aircraft.
Re:Lockheed Reassigns Workers to Fix F-35 Software by hosay123 on Thursday June 20, 2013 @02:33PM (#5913930)
> a system that is designed to never crash

I'm pretty sure no self-respecting engineer would ever attempt to design such a thing (although many might be deluded into thinking that e.g. advanced type systems might help achieve that goal).

> choice of using C++

Almost certainly not a good indicator.. much of those LOC are likely integrations from other projects. Who is to say the alternative didn't involve rewriting some more sensitive, costly-to-produce code in Ada?

Re:Lockheed Reassigns Workers to Fix F-35 Software by DanBC on Thursday June 20, 2013 @05:32PM (#5915103)
This website has some information about serious bugs. There are a variety of languages being used, by different industries, with either expensive war machines or expensive civil aeroplanes full of passengers or impossible to service space-craft.

(http://www5.in.tum.de/~huckle/bugse.html)

Re:Lockheed Reassigns Workers to Fix F-35 Software by johnchristopher on Thursday June 20, 2013 @03:33PM (#5914356)
Why did they switch from ADA to c++ ?
Re:Lockheed Reassigns Workers to Fix F-35 Software by smackfu on Thursday June 20, 2013 @04:34PM (#5914778)
Maybe all the ADA programmers retired.
Re:Lockheed Reassigns Workers to Fix F-35 Software by bediger4000 on Thursday June 20, 2013 @02:20PM (#5913831)
Any one who's worked at Martin Marietta in the past (now part of Lockheed) will recognize this. Martin had a long, long history of doing this sort of thing for projects in trouble.

When they get past this (and they will, they have the DoD over a barrel, they just have to not gloat too publicly) the fallout to the F-35 will last as long as F-35s are in operation. In the early 90s, Martin people hand-assembled Titan rocket guidance and control software because some time in the distant past, the assembler had some bugs. The F-35 software group will live with whatever horrific "best practices" the current group of reassigned workers come up with.

Re:Lockheed Reassigns Workers to Fix F-35 Software by masklinn on Thursday June 20, 2013 @03:20PM (#5914263)
> In the early 90s, Martin people hand-assembled Titan rocket guidance and control software

What the fuck?

Re:Lockheed Reassigns Workers to Fix F-35 Software by bediger4000 on Thursday June 20, 2013 @04:33PM (#5914770)
To be completely truthful, they compared binary dumps of hand-assembled, and machine-assembled software, to ensure that everything was exactly the same. But yes, they hand-assembled some weird instruction set. I can't recall asking if they assembled to ASCII hexadecimal word (24-bit) representations, or to something else.

I believe this was for "Commerical Titan", a short-lived, Titan 34D variant. Four flights, one of which left Intelsat VI in the wrong orbit. That wasn't software, but plugging the wrong cable in.

Re:Lockheed Reassigns Workers to Fix F-35 Software by SkyMarshal on Thursday June 20, 2013 @04:31PM (#5914758)
I'm hoping this will help justify limiting purchases of the F-35 and replacing it sooner with cheaper, more expendable, but roughly equally effective UCAVs and better missiles.
Re:Lockheed Reassigns Workers to Fix F-35 Software by jfb on Thursday June 20, 2013 @04:52PM (#5914879)
Common sense couldn't kill the utterly superfluous F-22; the F-35 has way more stakeholders, so don't count on it going away any time soon.
Re:Lockheed Reassigns Workers to Fix F-35 Software by AnIrishDuck on Thursday June 20, 2013 @02:14PM (#5913782)
Adding manpower to a late software project makes it later.

- Frederick P. Brooks, Jr. "The Mythical Man-Month".

Re:Lockheed Reassigns Workers to Fix F-35 Software by jere on Thursday June 20, 2013 @04:18PM (#5914661)
>Nine women can't make a baby in one month.
Re:Lockheed Reassigns Workers to Fix F-35 Software by macmac on Thursday June 20, 2013 @02:03PM (#5913709)
Not only have they added 200 engineers from other projects which will now suffer delays and quality issues, both also "They're running 24-7, what we call lights out." which will ensure massive burn out. Brilliant...
Re:Lockheed Reassigns Workers to Fix F-35 Software by sliverstorm on Thursday June 20, 2013 @02:44PM (#5914011)
If you time it right, you don't have to hit burnout. In my view, "timing it right" basically means completing the project before burnout. A successful completion followed by a little R&R goes an incredibly long way to combating burnout.
Re:Lockheed Reassigns Workers to Fix F-35 Software by macmac on Thursday June 20, 2013 @03:09PM (#5914175)
And I suppose they will use their awesome estimation powers that have led them into this mess, to time it right...
Re:Lockheed Reassigns Workers to Fix F-35 Software by sliverstorm on Thursday June 20, 2013 @03:17PM (#5914233)
Exactly ;)
Re:Lockheed Reassigns Workers to Fix F-35 Software by joezydeco on Thursday June 20, 2013 @02:23PM (#5913853)
Why is it called "lights out" if the offices are occupied with coders non-stop?
Re:Lockheed Reassigns Workers to Fix F-35 Software by thezilch on Thursday June 20, 2013 @02:32PM (#5913921)
Coder caves; no lights, especially not natural; time of day or passing is not in your peripheral.
Re:Lockheed Reassigns Workers to Fix F-35 Software by AlisdairSH on Thursday June 20, 2013 @04:21PM (#5914690)
Sounds lovely. Is that really a thing? I guess if you paid me enough money, I'd put up with that, but mostly, this makes me glad I declined their job offer several years ago.
Re:Lockheed Reassigns Workers to Fix F-35 Software by kevincennis on Thursday June 20, 2013 @03:29PM (#5914335)
"There’s nothing wrong with flying before we buy. In fact, most of us test drive cars before we [buy]"

Except that if we custom-order a car, the manufacturer isn't legally prohibited from selling it to someone else if we end up refusing to buy it.

Re:Lockheed Reassigns Workers to Fix F-35 Software by davorak on Thursday June 20, 2013 @04:01PM (#5914529)
There analogy is not good, but I would also assume that lockheed won a competitive bidding process to get the contract as well. They promised to build it at a certain price and in a certain time.
Re:Lockheed Reassigns Workers to Fix F-35 Software by kevincennis on Thursday June 20, 2013 @04:58PM (#5914924)
Yup, Lockheed's X-35 beat out a (horrendously ugly) Boeing airplane to win the contract, but my understanding was that the initial orders were cost-plus. That's pretty typical for large procurements of advanced aircraft. It's hard to know long it will take and how much it will cost to build something that bleeding-edge.
Re:Lockheed Reassigns Workers to Fix F-35 Software by sp332 on Thursday June 20, 2013 @02:08PM (#5913749)
The mythical man-month in action!
Re:Lockheed Reassigns Workers to Fix F-35 Software by jongraehl on Thursday June 20, 2013 @02:40PM (#5913979)
"Accountability". A conspicuously heroic effort (likely counterproductive long-run), because their +$6 billion drip is threatened by lawmakers who don't think they're taking this whole thing seriously.

For the sake of whatever competent programmers were already working on the system, I hope this is mostly PR noise.

Re:Lockheed Reassigns Workers to Fix F-35 Software by stcredzero on Thursday June 20, 2013 @03:49PM (#5914463)
That plane is not a replacement for the A-10, unless you kid yourself. The loiter time is atrocious.
Re:Lockheed Reassigns Workers to Fix F-35 Software by nether on Thursday June 20, 2013 @04:06PM (#5914566)
Because loitering and enduring battle damage make so much sense in an era of countermeasure-immune SAMs and huge directed-shrapnel warheads. Spitting DU rounds onto a city block also isn't a great idea in today's conflicts. The A-10 is obsolete, the tankbuster role is obsolete, the F-35 isn't perfect but is far more appropriate for today's conflicts.
Re:Lockheed Reassigns Workers to Fix F-35 Software by kbolino on Thursday June 20, 2013 @04:41PM (#5914830)
Yet we fight a lot of "obsolete" enemies. The effectiveness of the F-35 won't be known until it is battle-tested. Before then, it's all rainbows and unicorns.
Re:Lockheed Reassigns Workers to Fix F-35 Software by jfb on Thursday June 20, 2013 @04:48PM (#5914861)
We aren't likely to see masses of T-72s rolling under uncontested airspace any time again.
Re:Lockheed Reassigns Workers to Fix F-35 Software by harshreality on Thursday June 20, 2013 @04:43PM (#5914841)
Drones are far more appropriate for today's conflicts.
Re:Lockheed Reassigns Workers to Fix F-35 Software by vaadu on Thursday June 20, 2013 @04:54PM (#5914893)
The program should be cancelled.

It's way over budget, not as stealthy as originally claimed and completely unnecessary. The DOD should be putting this money and effort into UAVs.

I also have to wonder how quickly the Chinese will get the software updates.