News for hackers. Stuff that matters.
Re:The Pentagon’s $52,000 trash can by TrackerFF on Friday June 23, 2023 @09:25AM (#36446523)
I had the pleasure of being involved in a software project where the contractor was one of the usual suspects (Raytheon, Lockheed, etc.).

Prior to this, I had no idea why defense projects were so expensive and slow. You have to understand, there are so many levels of negotiation, over the smallest changes imaginable. There's no such thing as "open source code" in these projects, everything is either modified off-the-shelf, or completely boutique stuff, and of course implemented in such a way that the clients (that would be us) can't just access the code themselves, or that clients can just build upon or make wrappers.

The contractor will send you their own techs or engineers - even if it's across the world - to fix the most trivial bugs. And it will take months or planning, budgeting, and what not.

In once instance, one contractor tech spotted a bug in the code, which some user had reported about - it was a one character fix, just a typo, really. So he fixed it on the spot, along with the rest of the stuff he came for.

A couple of weeks later he comes back, and reverses the bug. He had told his manager about the bug fix, and was berated for fixing in. In order to fix that bug, it had to be reported correctly, go through software update meetings, go through budgets, and the whole nine yards.

The bug was reported the correct way, and many months later he came back to fix it (again).

The whole process of specs and updates was like pulling teeth.

Re:The Pentagon’s $52,000 trash can by giancarlostoro on Friday June 23, 2023 @09:27AM (#36446557)
> A couple of weeks later he comes back, and reverses the bug. He had told his manager about the bug fix, and was berated for fixing in. In order to fix that bug, it had to be reported correctly, go through software update meetings, go through budgets, and the whole nine yards.

What do you mean you went and just fixed it!?

We need all of middle management to charge for the fix!

Re:The Pentagon’s $52,000 trash can by vegetablepotpie on Friday June 23, 2023 @10:04AM (#36446969)
My experience has been that you can use open source software, but security considers that to have a high risk and you need to make a business case for it.

What isn’t a security risk? A closed source vendor supplied application that has its headquarters in the US.

You can sometimes find pockets of innovation in the defense sector, but it’s mostly filled with bureaucrats with engineer in their title.

Re:The Pentagon’s $52,000 trash can by ajmurmann on Friday June 23, 2023 @10:35AM (#36447343)
To be fair, securing the OSS supply chain is a tough task, especially when pulling in new versions.
Re:The Pentagon’s $52,000 trash can by xxpor on Friday June 23, 2023 @11:38AM (#36448374)
And yet, things like the Patriot missile system get built. It's pretty wild what an 800 billion dollar budget will paper over.
Re:The Pentagon’s $52,000 trash can by david422 on Friday June 23, 2023 @10:05AM (#36446982)
So I worked for a company that had loads of process for even the smallest changes. It was frustrating, but what I came to realize was that the company was not interested in fixing bugs or building features - they were primarily interested in not breaking things/keeping the status quo. This was an established business that was concerned with keeping the business running, not chasing the latest and greatest. Perhaps a lot of it was just CYA, but also means it was pretty hard to "move fast and break things".
Re:The Pentagon’s $52,000 trash can by DropInIn on Friday June 23, 2023 @10:09AM (#36447036)
That sort of business sounds like what we want in milsec and government suppliers, no?

Reliable even down to the bug behaviors...

Re:The Pentagon’s $52,000 trash can by ajmurmann on Friday June 23, 2023 @10:39AM (#36447410)
The reliable bug behavior is an underappreciated point and can be hard to understand. I work on software that gets deployed on-prem by customers and had very powerful capabilities for deploying your own code to it etc. There have been a few discussions about some bug fixes and in particular security fixes being appropriate in a patch release when following SemVer. Something might be unintentional or insecure, but customers might be building on the broken behavior and now deploying a patch breaks their apps. The line can become quite blurry.
Re:The Pentagon’s $52,000 trash can by pc86 on Friday June 23, 2023 @11:19AM (#36448066)
This sounds like more a deployment/release process problem than anything else. If the customers are essentially deploying untested code straight to production, they're doing something wrong. It might not be their fault, maybe they don't know they should have a staging environment first, or maybe your pricing model makes it prohibitively expensive. It's not as simple as "your customers are doing it wrong," but if you upgrading a package breaks their production environment, someone somewhere is - or more likely, many people are - definitely doing something wrong.
Re:The Pentagon’s $52,000 trash can by danbruc on Friday June 23, 2023 @11:45AM (#36448471)
It is not necessarily about breaking the production system, just breaking the system because it depends on some bug or contains workarounds that break if the bug is gone. Even if it breaks before production it is still broken and might - depending on how old or central the bug was and how much code was build on top or around it - require substantial effort to fix.
Re:The Pentagon’s $52,000 trash can by pc86 on Friday June 23, 2023 @04:53PM (#36452040)
100% agree it just sounded like in this instance the first time they'd hear about it breaking a customer's workflow[0] would be in that customer's production environment. I've been on both ends of solutions where customers are either technically or financially incapable of having robust test environments for even critical software, especially if they're relying on someone else to write it.

[0] https://xkcd.com/1172/

Re:The Pentagon’s $52,000 trash can by birdyrooster on Friday June 23, 2023 @11:56AM (#36448648)
Yes this happens all the time, I’ve seen it everywhere I’ve every been an engineer. Bug fix roll back due to causing an outage because it broke the defacto API being depended on.
Re:The Pentagon’s $52,000 trash can by fivre on Friday June 23, 2023 @12:43PM (#36449286)
I have had this concern raised, and I'm unclear how, if you adhere to that model of compatibility, that you don't effectively dispense with semver altogether and make every release a major version bump, as every change is breaking if you relied on broken or insecure behavior in earlier versions.

IMO it makes more sense to communicate the expected impact of the changes, so that downstream can read a patch bump as unlikely to cause issues.

Re:The Pentagon’s $52,000 trash can by ajmurmann on Sunday June 25, 2023 @11:47AM (#36469469)
Yeah, it gets fuzzy. Unfortunately, major releases are expensive in that support contacts are typically guarantee support durations for major or minor releases. Having dozens of versions that all require separate patches quickly can absorb much of your orgs productivity.
Re:The Pentagon’s $52,000 trash can by deelly on Friday June 23, 2023 @10:36AM (#36447364)
Not only. Also some software for financial sector, for suppliers, for maintenance too. Basically, anything that do not crash a system could be a feature. And sometime even have logical and good reason to exists.
Re:The Pentagon’s $52,000 trash can by ben_w on Friday June 23, 2023 @10:47AM (#36447532)
I imagine it depends on the specifics. Closest I got[0] to the defence industry was all the anecdotes my dad had about Plessey and GEC Marconi.

I'm sure it's important to have a stable technology to train on in the military; at the same time, I remember one of their open-days they were showing off a civilian in-flight entertainment system that was so clearly overpriced for what it was that I knew even as I saw it that I could've done better for lower unit-cost. This isn't to show off, even despite my having not yet finished secondary school at the time — my solution would've been little more than "glue laptop to chair/wall in front", as tablets weren't something people discussed much around me in the 90s — it's just that the business solution was really unimpressive.

I assume there's secret military equivalents to that public project.

[0] if you exclude the job interview at Lockheed Martin for the year in industry paid work experience part of my degree, which didn't go further than an interview

Re:The Pentagon’s $52,000 trash can by DropInIn on Friday June 23, 2023 @07:10PM (#36453487)
If they just buy whatever is most cost affordable then the risk factor for trojans goes sky high. Even with major brands this would be a significant issue (see HDD manufacturers bs with drives shipping with malware in the firmware).

Theres a reason there's so many steps to accept hardware.

Re:The Pentagon’s $52,000 trash can by schemester on Saturday June 24, 2023 @01:35PM (#36460816)
No, of course not, if your military doesn’t develop tech faster than the competition then you lose.
Re:The Pentagon’s $52,000 trash can by nmca on Friday June 23, 2023 @11:35AM (#36448321)
I too worked for Google once.
Re:The Pentagon’s $52,000 trash can by rurp on Friday June 23, 2023 @02:59PM (#36450888)
That kind of slow and steady approach actually sounds pretty reasonable for some cases. Sure it won't win the market for a trendy new consumer app, but a lot of software is basically feature complete and should prioritize stability over whatever's currently cool and exciting.

I have an old account with a bank that has hardly changed its website design in well over a decade. It's great! The information is clearly laid out in tables and everything loads super fast. Every other financial web app I'm familiar with has a much more "modern" design and is categorically worse to use. One site literally takes over a dozen clicks to get to basic information and evey single page load is slooooow.

Re:The Pentagon’s $52,000 trash can by pradn on Friday June 23, 2023 @03:49PM (#36451423)
Bank webpages don't need to be more than HTML tables, with perhaps a bit of JS thrown in to ease element collapsing, etc.
Re:The Pentagon’s $52,000 trash can by numpad0 on Friday June 23, 2023 @10:11AM (#36447067)
But you'd be just as upset as his employer if you'd come across a binary release built from one commit ahead the GitHub repo. The absurdity is in the amount of overhead, not the fact that there are overheads at all.
Re:The Pentagon’s $52,000 trash can by 2OEH8eoCRo0 on Friday June 23, 2023 @10:22AM (#36447186)
I worked on a radar system that could see precipitation- think doppler radar map view. The ability to do this was in the contract but there was no way to view it. We had Matlab scripts so we the devs could view it and the weather people wanted them. We weren't allowed to hand them over without a contract. They'd pop in multiple times a day to view our doppler.

The adversarial nature that defense companies treat our military is appalling. I'm convinced that refactoring a lot of this BS could get us way more with less $

Re:The Pentagon’s $52,000 trash can by mortify on Friday June 23, 2023 @01:51PM (#36450115)
If the viewing software was provided, the support would be included and that will cost money in the long term. That is something that the company didn't agree to provide nor the government to pay for. In this case, the adversary wasn't the contractor, it was the users in the government and the government employees holding the purse strings. No contractor will turn down the chance to extend the contract.
Re:The Pentagon’s $52,000 trash can by JamesBarney on Friday June 23, 2023 @05:04PM (#36452138)
A lot of this adversarial nature comes down to how projects are awarded. Government employees have very little discretion to choose projects, so they always go with the cheapest bid. But no large software project ever has had 0 change orders.

So the only profitable way to win these contracts is offer the cheapest bid by delivering as little as possible while being in spec, and then pillaging on the change orders.

Re:The Pentagon’s $52,000 trash can by FooBarBizBazz on Friday June 23, 2023 @05:09PM (#36452186)
What about the "new defense" companies, who use VC money to build demos? Surely they eventually want to build moats and sell razor blades, but for now what they're doing seems different?
Re:The Pentagon’s $52,000 trash can by jliptzin on Friday June 23, 2023 @10:35AM (#36447341)
I have encountered a situation a few times where a bug that went unnoticed for a while was actually keeping another, more serious bug from surfacing. So when the first bug was fixed, it caused more serious problems elsewhere in the system. I could see where they're coming from where any kind of "fix" needs to trigger a careful reevaluation of the whole system. That being said, I don't see a need for the Kafkaesque months of planning and budgeting that follow.
Re:The Pentagon’s $52,000 trash can by sneak on Friday June 23, 2023 @11:12AM (#36447955)
> That being said, I don't see a need for the Kafkaesque months of planning and budgeting that follow.

It's a method for transferring billions of dollars from public tax revenues to private companies which are cozy with the state. It has nothing to do with safety or reliability.

All other methods of stealing tax money are illegal.

Re:The Pentagon’s $52,000 trash can by jliptzin on Friday June 23, 2023 @12:01PM (#36448719)
Would be nice if we spent some time cracking down on that kind of tax theft rather than wasting time on drag shows and the 3 dozen teens in the whole country on puberty blockers
Re:The Pentagon’s $52,000 trash can by red-iron-pine on Friday June 23, 2023 @12:20PM (#36448994)
you got it mon ami. that's the whole point of making a kerfuffle about drag shows. or gay marriage. or even abortion.

ain't no one talking about tax cuts, or getting after the rich, cuz of that.

Re:The Pentagon’s $52,000 trash can by syntheticcorp on Friday June 23, 2023 @01:25PM (#36449805)
I’ve also encountered that a few times where a fairly anodyne bug in a codepath prevents a serious security bug from being reachable. With my attacker hat on it is very tempting to just report the first one…
Re:The Pentagon’s $52,000 trash can by bentxt on Friday June 23, 2023 @11:14AM (#36447977)
Do you have an opinion whether this kafkaesk bureaucratism is especially well established in the domain of software or is it a broader pattern? Where does this come from?
Re:The Pentagon’s $52,000 trash can by renewiltord on Friday June 23, 2023 @11:44AM (#36448458)
Remember: Regulations are written in blood.
Re:The Pentagon’s $52,000 trash can by AndrewKemendo on Friday June 23, 2023 @12:05PM (#36448781)
It’s a broken system for sure…

HOWEVER when your acquisition system is designed to ensure that a nuclear detonator and a LAMP application go through the same process, these kinds of “overkill” or “nonsense” requirement responses happen.

We’re long overdue for fixing the FAR and federal acquisition process, which is the root problem. The legal system and processes built over the last 50 years, and as a result the people inside it, doesn’t allow for simple and efficient differentiation between projects legally so you end up with the same hammer meets mail approach.

Re:The Pentagon’s $52,000 trash can by Eddy_Viscosity2 on Friday June 23, 2023 @01:02PM (#36449534)
Modern defense contracting has become, first and foremost, a process to transfer government funds to private hands. The fact that any useful products or services happen is purely incidental.
Re:The Pentagon’s $52,000 trash can by quantum_state on Friday June 23, 2023 @08:06PM (#36453919)
you can say that again
Re:The Pentagon’s $52,000 trash can by Solvency on Friday June 23, 2023 @06:04PM (#36452753)
Ok but...why. Why is it like this.
Re:The Pentagon’s $52,000 trash can by stonogo on Friday June 23, 2023 @08:13PM (#36453972)
Part of this stems from government accountability issues. If you fix that bug, and you don't go through the process, no you didn't. At any time the government can ask for a contract performance review, and doing things the easy way means lots of things that should be tracked don't get tracked... which means they never make it into the contract performance review.

They mean well, and I understand how hard it can be for government agents to hold massive contractors accountable, but the purpose of a system is what it does... so we wind up with entrenched bureaucracy, and the players who don't subscribe to these approaches wind up getting outcompeted because their on-paper performance doesn't measure up. Even if they're in fact a million times easier to work with and do a great job.

Re:The Pentagon’s $52,000 trash can by d-man on Sunday June 25, 2023 @02:28PM (#36471199)
s/defense/regulated industry/ I found a bug in my company’s software product once. The code was using the user’s display name in a lookup rather than the username. I grabbed a QA engineer, he agreed it was a bug, and he put the wheels in motion. I don’t think that fix ever made it into a release.

Our software was part of the stack that our regulated customers were using to document their compliance. Every year or two, they would audit our development process to insure that untoward code wasn’t slipping into the release. Getting the fix into a release would have required that the bug be prioritized, fixed and qa’ed. Never became important enough. It burned me at first, but I’m not sure it was the wrong outcome.