In a nut shell, here's why things are so screwed up IMHO:

1) Most of these companies have had audits, but they're being done by 3rd rate or very inexperienced external consultants.

2) The companies limit the scope of the tests. Real hackers don't give a shit about your scope of work, they have no rules, only goals.

3) Even when a test is properly done the exec management looks for silver bullet product solutions instead of changing across people/process/technology

My company solves #1, but we can't do anything about #2 or #3 :-/

Add some funding / IT cost center, no value add language in there as well.

This is very likely not the full story, unless the 0day in VSA was somehow wormable. That "deployment" is doable through overly permissive IAM and everything else that enables privesc.

There are two parts to these vulns. Whatever gets the foothold, and whatever allows privilege escalation. Audits do a great job in catching the misconfigs that allow privesc.

The tragic thing about these attacks is often the blast radius can be contained fairly easily by asking the right questions... If you're someone who has passed these audits, or done these audits, it becomes pretty easy to see how many unforced errors go into these catastrophic attacks.

Without seeing the codebase in question, you can't be sure, but having been a web app pentester for 10+ years, these are the kind of issues that were found regularly, and whenever I saw classic ASP in tests, they were the kind of issues I'd be looking for, knowing the inherent weaknesses in the platform.

The only way to get secure software is to increase liability of parties involved.

My suggestion: Start with the ability of any customer to return any purchase (hardware/software) which contains software with a disclosed but unfixed CVE after 90 days without a patch. If this doesn't get rid of the Internet of Shit, I don't know what will.

Next, set a minimum damage rate of 100 USD per user for each data-breach that involves personal information and 1000 USD for any special kinds of personal information (credit cards, etc) and 10000 USD for any protected health information breach.

To include the liability of the attackers, which I think will ultimately be more scalable and effective than punishing the victims. Not saying there aren't incentives for the victims to "do better", but I think that will only get us so far.

History paints a picture of societies evaluating the effectiveness of better walls vs. owning the landscape, and deciding on the latter as being a more beneficial approach. It's how we get the saying that "Rome conquered the world in self-defense". I would bet that's where this ends up after enough material losses.

Does this not also just kill tech? CVEs pop up decades after products have died. Now every tech product is just one unsupported CVE away from losing _all_ lifetime revenue. I just can't see how anyone would ever invest again...

edit: to clarify further the fact that any CVE triggers this, no matter how small, seems egregious to me. The idea of there being no lifetime on the liability seems wild given how CVEs are often the result of other developers breaking ABIs. Imagine a profitable product that was last sold 10 years ago having it's full lifetime revenue refunded because of some change in glibc.

Are they even covered within the warranty period? I never tried it, but I think I'd have an interesting conversation if I went to a shop and told them I want to return a product because while it works flawlessly it's got a vulnerability. The standard procedure is usually getting a replacement, but this isn't possible here as the whole product range is affected.

I had played around with the idea of requiring support for 3/5/7/10/whatever years after the cessation of sale, kind of like how car manufacturers are required to offer parts support for 10 years after sale, but I can see that causing enough overhead that many tech devices simply would never get made.

I think you underestimate the (a) greed and (b) capabilities of the people involved.

- How about 5 years minimum for hardware? And as much as the vendor wants to promise.

- How about that requiring that vendors at least allow for customers to pay for extended support for another 5 years by paying 20% of the initial price per year.

It is just ridiculous that currently many devices are insecure 3 months out of the gate.

We already know its neigh impossible to ensure software is bug and vuln free. We have to reckon with the fact that secure software is possible, but extremely hard and impractical to achieve with any regularity.

Because of well known computer science problems such as the Halting Problem[0] and more that I am totally ignorant of, the only alternative is extremely thorough verification like happens in planes[1]. The rub is that unlike aerospace and other critical controls, we cannot define the software security problem as closely as is possible in those other critical systems. Doing so will undo the benefits of our general technology solutions.

The reason I claim it is nonetheless not a software problem is: In case after case of these troubling examples of security failures, there are only a few organizational commonalities that link them all together. Time after time, there are shocking misconfigurations, corners cut, best practices eschewed, warnings ignored, processes disregarded or absent and swept under the rug.

C-Suite execs, as has been well noted, opt for the checkbox-style silver-bullet-whiz-bang, because they can understand the value proposition of one-problem-one-fix and it is easy to communicate to stakeholders. They are totally ignorant to and uninterested in the details of actually providing a quality product. Their product is often not what they are selling; They are getting their bonus, and they are gonna exercise those stock options.

The only way to make a dent in this security problem is to require adequate processes for infrastructure the way we require adequate lighting, or fire protection devices, or plumbing. Until it is a business requirement to provide appropriately managed processes, infrastructure, testing, and finally development practices to go along with it all, there is no hope.

Will your school or hospital be shut down next? Will it be your bank that is defrauded when your cardholder details are skimmed at the local gas station or stolen from a multinational retailer? Will it be your government's top secret clearance database, your municipal water, your child's photos from their phone or social media?

Our only real hope is that the next victims of this heinous, Steinbeckian, tragedy are the Congress, Senate, and whoever else it takes to get a fucking grip.

[0] - Computerphile https://www.youtube.com/watch?v=macM_MtS_w4

[1] - Examples of aerospace validtion https://scholarworks.sjsu.edu/cgi/viewcontent.cgi?referer=ht...

Having classic ASP pages hosted on a production system in 2021 seems like a pretty strong indication of a lack of codebase maintenance and auditing.

AFAIK classic ASP pages can be as secure as those in any other framework. The vulnerabilities (most commonly SQL injection) are known and are addressable.

I know it chafes at some (Microsoft marketers esp.?) that ASP pages are still around. But classic ASP is yet another example of that old adage "If it ain't broke, don't fix it!"

I watched one organization assign an entire division of programmers to develop a moderately-sized ASP.NET application: they went through orientation and training in ASP.NET and then planned, designed, coded and rolled out ...nothing! After two years there was literally nothing!

A perceptive manager in another division approached his sole ASP developer and asked if she could write some ASP code to "demo" that same project. She looked at the specifications and then quietly wrote an entire system!

Weeks later, when the department heads saw her "demo", they thought the ASP.NET developers had completed it. Imagine their surprise to find that what they were looking at was done, not by the ASP.NET group funded millions of dollars but by someone in another division: a single focused classic ASP developer quietly working downstairs.

Nonetheless IIRC ASP pages reach end-of-life support by Microsoft in 2025 so it might make sense to migrate. But b/c classic ASP was written in a manner consistent with early web standards (CGI/APACHE) migration of classic ASP to one of the various classic ASP-like frameworks in PHP, Perl, Ruby et al would likely be easier, faster and cheaper. Migration would be mostly translation and much, if not all, could be automated.

In contrast, moving classic ASP to ASP.NET would be more fraught with problems. The underlying models of the WWW are inconsistent.

Also, how many classic ASP developers are there now , to add new protections for new issues...

And if you look at the linked thread, it doesn't seem like they've done a great job of maintenance...

Yes, it is just like PHP or Perl or Ruby with the CGI specification.

raesene9 says >" how many classic ASP developers are there now ..."<

Good question! But IMO a web developer could learn classic ASP in a single day!8-))

raesene9 says >"the individual developer has to code all defence,<

Yes, just like PHP or Perl or Python.

or.... they could move the application to a supported platform with a wider developer base and in-built protections against common web application security attacks.

There is a reason why most web app developers do not hand code their entire stacks. For example why ruby web developers use rails. Web app. sec. is a a relatively complex field to get right, full of edge cases. Having each and every web app development team take the time to understand all those cases and edges probably isn't the best use of scarce developer resources.

Yes yes. And this was of course very maintainable, up to todays standards, easy to onboard new hires in, not falling apart in edge cases that weren't part of the demo, code that was audited by multiple developers, it didn't make any shortcuts in terms of security, etc. etc. etc.

That some business guys are amazed by the "lone wolf dev skills" might be explainable, but on HN we should know better. Yes, there are devs that get a ton done irrelevant of framework, but "getting things done" (in terms of business requirements) is only part of the story.

Eventually, after many months and at least a couple million dollars, I and three other people got dragged in: another developer (no one lets me do UI code), a good project manager to coordinate with users and management, and a developer/tech lead/manager who had two unique abilities: good at cutting Gordian knots, and having enough political capital to say "no" to people (most people around couldn't say "no" to save their life; I can and do, but nobody listens to me). The latter promptly booted the previous team off the project.

Within four months, we had the new app up and running and in production. I'm told that it has one of the best issue track records in our area, including never having had a sev 1 outage. On the other hand, I have heard some of its current maintainers complain about it not being "up to todays standards" because we deliberately kept it simple rather than adopting a bunch of complexity for resume-padding reasons.

I'm less cynical than you are about the parent comment, largely because I'm watching the same scenario unfold within my own company.

We're on year two of the million-dollar team turning out nothing. I'm in a different department, but my position means I liaise with lots of different people throughout the organization, so I also know that a single dev in a third department is half-way through solving the problem because a ultra-high-level manager doesn't want to wait anymore and will use it as an excuse to can the other team and that department's director.

And this was of course very maintainable, up to todays standards, easy to onboard new hires in, not falling apart in edge cases that weren't part of the demo, code that was audited by multiple developers, it didn't make any shortcuts in terms of security, etc. etc. etc.

Unless you've seen the person's codebase, it's uncharitable for you to assume that it's deficient, perhaps based on your own experiences.

If this scenario can happen in two companies, it might be more common that any of us realize.

What was shown wasn't a demo - it was a full implementation. It was rolled into production weeks after the first showing and AFAIK is still in use 20 years later.

What is there other than "getting things done" (that is, other than bitching about it)?

That being said, we should really take these anecdotes with the appropriate grain of salt. After all, we're here in a thread about countless companies being threatened by security flaws. A dev that "gets things done" (from a business point of view) might actually do that, but do they also think about all those other requirements that business themselves can neither validate nor appreciate if measured by short-term KPIs?

I'm not saying that's happening, but I've seen too many "Why are you spending weeks on this, I can do this in half a day!"-kind-of-devs, that then hack something together that technically works as the requirements demanded, but completely falls flat under any aspects of maintainability and extensibility, let alone security.

No, you shouldn't be surprised about this, it's natural that smaller dev teams are actually faster. This is Mythical Man-Month.

But the best way to do this is (and the way consistent with this thread's narrative) is to to develop the final system before you tell the CEO. Oh, and best to have a supervisor for you who does the telling.

Old and well maintained software can be reliable and secure, it just rate to encounter it. Maintenance is underappreciated. People don't get rises and promotions for successful maintenance of an old system. But they get it for new projects even if this project is rewriting of an old system in a new language/framework (even if a rewrite introduces new bugs, vulnerabilities and drops some old features).

So if an organization has money to spare, the software will be re-written every several years to flow the fashion and if doesn't have money security will suffer too.

The problem is that in most codebases I've seen, once you get to 19 years passed deprecated, it's not that it's a well maintained hardened environment. It's a poorly maintained environment that people don't want to clean up for fear they break something.

In theory you can totally add those protections per application but the effort of doing that, maintaining the knowledge required per application or team and keeping on top of new research, is likely higher than just moving to a new framework which has in-built protection.

Also you have to consider developer availability. At 19 years since it was deprecated, there is a smaller pool of people who are skilled at maintaining that codebase, and the group of people who can do that, and keep on top of web application security attacks is even smaller.

Though Classic ASP was released much earlier in 1996 and I don't know if SQL libraries offered parameter binding and template engine - escaping of string in template variables.

I started in security in 2000, as an analyst for an org using classic ASP. maintaining security was a pain.

I then started as a pentester in ~2005 and what I can tell you is, in my experience, classic ASP applications rarely had good protections against injection attacks (e.g. XSS, SQLi) for precisely the reason that the framework did not protect against it, leaving developers to make sure they had routines to canonicalise/sanitize/parameterize input correclty, and also that they implemented them universally across all possible user inputs.

Whilst this isn't impossible, in my experience, it was rarely done perfectly.

In comparison, something like ASP.Net which had in-built protections available, at least had the chance of having good uniform protection.

Or we show that we can deliver safely and resist business pressure to deliver fast and cut costs, or the government is going to do it for us.

People just don’t take virtual things as seriously, unless they involve conspiracies.

*edit: when I say "people", I mean the end-users who would otherwise demand change.,

Everyone who wrote it is retired or dead.

There is significant complexity and depth to front end tooling these days, and I consider my colleagues working in that space to be as talented and experienced as anyone else.

The issues in the frontend space seem to me to arise as an artifact of the fact that the work they produce is far more visible and directly evaluateable by non-technical people. The demands they have placed on them to deliver features quickly is far higher. They're far closer to the user-experience side of the product than most backend devs. And they have to deal with a tools ecosystem that evolves and changes far faster than others.

Trying to describe that complex issue in terms of stratification of developers into "serious engineers" and "not serious ones" does a disservice to the underlying problem, and doesn't help address it.

The flux and frenzy in the space is much more a sympton of the novel nature of the application architecture, the size and speed of growth of industry around it, and the rate of experimentation with frameworks and tools to quickly build extensive applications within it. Those are not self-inflicted, but circumstantial.

> That’s a separate issue from how smart people working on the issue may be.

I took particular issue with the original commenter dismissing an entire class of developers on the basis of what tools they worked with. I found it to be an example of gatekeeping that unfortunately is far too common in the industry.

https://en.wikipedia.org/wiki/Equifax

https://en.wikipedia.org/wiki/List_of_data_breaches

https://nvd.nist.gov/vuln/search/results?form_type=Basic&res...

Your point is well taken, even though things are not as bad they are still bad.

I sound like a broken record but it bears repeating: Most of these attacks are successful because companies neglect best practices. Whitelisting, security awareness training, UEBA, etc go a long way.

I would hope the free market would prune companies without proper cybersecurity but regulatory capture means it probably won’t. Equifax and its executives are doing just fine.

Three things.

1. Markets have a slow reaction function, and it really is a reaction function. Let's Consider Equifax. Suppose that market were competitive. Then you'd have dozens or even hundreds of firms with all of that data that Equifax leaked. It seems unlikely that having more copies of data floating around more firms would decrease the risk of a breach. By the time the market signals, the damage is already done, and Equifax going out of business does diddly squat for impacted consumers.

2. Markets also have perverse incentives. Data breaches, in particular, are not necessarily expensive. I've been affected by at least a dozen, none of which had a material impact on the company that lost my data. None of those companies except equifax is subject to any sort of monopolistic forces. Some, like dropbox, are basically commodities. This might be different in the case of Kaseya and Solar Winds, which are effectively IT security outsourcing firms. Maybe. We'll see. If both of those firms continue to exist at similar scale, then the hypothesis that markets can do literally anything about IT security is completely discredited.

3. Equifax is definitely a monopoly/triopoly, but the situation is much closer to cartel behavior than regulatory capture.

Large businesses can trivially set aside a few million dollars per year for ransoms. Small businesses don't have the ability to change the system.

Society expects technology to "move fast and break things", at least until the consequences get too visible. And we're not there yet.

Much as developers in the past couple of years have really started to grapple with the fact that as wonderful as software libraries are, dependencies have non-zero costs that come with their benefits, connections need to be seen as being having non-zero costs as well. I work in a similar situation myself and I know my philosophy over the past 10 years has very much gone from a permissive "well maybe I'll need this later" open-ended protocol design to a super-strict "this connection moves exactly this set of strongly-typed, verified messages over it and if anybody tries any funny business we slam it shut and scream bloody murder on the monitoring" philosophy. Especially challenging with the Web's support for messaging, which is great for being a web page but is flabby and bloated for a messaging solution, what with all these headers that do magical things in the web servers or anything in between.

As our systems get more and more complicated, biological metaphors seems to be ever more useful, and I think we need to look to nature's highly defensive "programming" a bit more for inspiration. Chemistry has some different characteristics from our programming environment, some to nature's advantage and some to its disadvantage, so we can't blindly copy it, but living things take 'message security' very seriously. You don't last long if you just blindly trust everything out there.

I don’t know how you coordinate this without regulation and something like P.Eng type certification (which comes with it’s own problems).

People without credentials but with experience and care can produce a reasonably secure system. People with credentials but under time pressure and contradictory requirements, and willingness to cut corners, can produce an unsafe system.

It's the system that fails. It can be inspected the way buildings are inspected.

If we required something like a P.Eng, which (at least theoretically) means you are responsible with real consequences (for say bad building design) then certified employees will be far more reluctant to attach their name to shoddy practices. This bubbles up to the employer. If a client requires you to have certified P.Eng sign off then there’s some reassurance it’s not just going to fall a part.

I don’t think you need this for everything in a stack, but security wise it might be a good idea.

The problem there is that everything on the stack is a potential hole. The chain is only as strong as its weakest link.

We accept that possibility in SaaS though... for now.

Most successful SaaS applications probably weigh in at 250k LOC or more (I know LOC isn't a great metric, but it serves as an idea of an application's complexity) and will use many libraries that themselves could be questionable. Seems like it would be pretty complicated to certify.

It's far less the programmers than the business plans that demand minimal investment. These are classic externalities that serve to damage society at large.

Think of all the security theater in airports, for instance. Think of all the to and fro in many policies as different officials get elected every few years.

I’m not sure what the answer is but better security and a rethinking of user authorizations seems to be in order.

The key to the current spree of ransomware is the massively improved ability to monetize digital hostage-taking. I don't really understand how financial watchdogs have let this go through, but cryptos have become a massive loophole in kyc and anti-laundering regulations. Recent moves in that sector seem to hint that this party is about to end and, hopefully, will create enough friction to reduce ransomware activity.

There's actually less illicit activity in cryptocurrency than in USD.

> The amount of environmental damage caused by proof of work is massive.

Not even 1% of global energy consumption. When is this FUD gonna stop?

Which... it already has. Coal plants have been spooled up just to mine btc. The idea that we should continue or expand crypto mining is ludicrous, because it already has harmed the environment, even at its small market cap.

The real question is: why do we still allow oil companies to exist when they're the ones responsible for much of the world's pollution? Because the USD is backed by oil.

Probably because attempting to enforce an oil ban overnight would probably involve death counts in the 9-10 digits. That is why any attempt is going to be progressive, and we can all agree that it's not going nearly fast enough.

That question is kind of a diversion from the current discussion, though.

It's really not. Cryptocurrencies have no real impact on the environment so discussing that is not really productive. Better to redirect discussion towards real problems which are actually destroying the planet. Problems which will never be properly solved because powerful people depend on their existence. The petrodollar, trade with China, etc.

> It's really not. [...] Better to redirect discussion

At some points the strings are getting too visible.

People insist on posting this FUD. I get it, they're concerned about the environment. So let's talk about real problems instead. Such as the highly polluting chinese manufacturers which no doubt fabricated the hardware we are using to write our opinions.

The real distraction is this "cryptocurrencies kill the environment" idea.

People should be able to transact freely without some government demanding explanations. If there's more crime as a result then so be it.

This discredits your entire post. Your other points may be valid, but how can anyone be sure if this one is provably false?

My domain registrar does, and it's not exactly a fly-by-night operation.

I think at least one airline does. Though I might be remembering that wrong.

I won't ever get into crypto because I like money that keeps working when the lights go out. But I don't think it's either as fringe, nor as mainstream, as the two sides present.

https://www.bbc.com/news/business-57096305

I personally would be surprised if Tesla made more money from selling those cars than they made by pumping btc with that news line.

Of course, the internet would lose its mass appeal. Maybe it wasn't meant to be.

To me, this is more of a foreign policy issue. I’d say the damages caused by an attack like this can be requantified in loss of American life, and treat it like such. What should we do if Russia were killing 20 US civilians every few weeks?

It just costs a very significant amount of money. Many businesses just don't see the lowered risk to be worth the expense.