There's a very "child-like" (not in a good way) form of responsibility that everyone seems to lean into as they climb up - very intent-based.

I asked them to do a thing, but didn't intend the obvious consequences* so it's not my fault they occurred.

> I asked them to do a thing, but didn't intend the obvious consequences* so it's not my fault they occurred.

And now we have the same thing but the bosses 'hire' AI.

Now I realise this is part of how unusual my thinking is.

I'm happy to use phrases like "ChatGPT hacked out of the sandbox, then hacked into HuggingFace"; people often respond to this like I'm suggesting OpenAI isn't at fault, and like, that's not my position at all, so far as I'm concerned the buck still stops with the person who set the task regardless, the thing that changes from incidents like this is now nobody in the future gets to even have the excuse "oh but we didn't know it could even do that" or "we didn't know it might interpret our orders in that kind of way".

The response, both when a human messes up and now when an AI messes up, needs to be defence in depth: someone giving orders needs to be giving clear orders, entities (human or machine) who follow instructions need to have not just an understanding of how to follow them, but also what's so out of scope as to be forbidden - the difference between 'follow orders' and 'follow lawful orders'.

You have a very engineer-like approach, like if you draw the line from A to B everything will work fine. Real world is different though. Humans will blissfully ignore the orders, business analysis is a lost cause since decades, and AI is built on human knowledge so guess what it will keep doing. Now what? How do we build systems without assuming complete adherence, but tolerating imperfection and failures? Isn't there some discipline teaching us that?

I indeed have a engineer-like approach, but engineering absolutely does not assume draw line from A to B and expect that's enough:

Real world, as you say, not so simple. Everything has to deal with certain degree of forecastable nonsense, e.g. a bridge has to cope not only with traffic and winds, but the possibility that someone will be drunk in charge of a ship and crash into it.

> AI is built on human knowledge so guess what it will keep doing.

Yes, and also brings its own additional mess on top of that. All machine learning takes a huge number of examples to get good, so an LLM isn't just "read all the online courses in how to run a business", but also likely has 50 business versions of the recent demonstration of common sense failure with "I live 100m from a car wash, should I walk or drive?"

> How do we build systems without assuming complete adherence, but tolerating imperfection and failures? Isn't there some discipline teaching us that?

Many such disciplines. Perhaps all except maths and computer science? Or even including maths and computer science, given stats is part of maths and even compsci has to deal with fault tolerance.

> Isn't there some discipline teaching us that?

That’d be engineering.

Yes and there is a reason, we are in theory, held to such higher standards. I used to think it was harder to draw the line with AI. The simple reason being when I as an engineer say something is not feasible (not not possible, just wrong and not worth doing), I could be overridden with AI. Now that we have more complex models == more money, I can draw the line with dollars and that is a language way stronger than technical feasibility to management.

This is literally the deepest I've gotten into a thread before AI was blamed, for MONTHS.

Massive reorganisation will only happen if companies with poor security record go out of business, while competent ones win market share.

Otherwise shareholders do not care, because they do not have skin in the game.

Same for government staff. Unless they are explicitly fired there are no consequences of abusing the trust of public.

> Massive reorganisation will only happen if companies with poor security record go out of business, while competent ones win market share.

... said every "security" pedlar ever.

Yeah that is a pretty weird opinion. Who cares about if he gets fired. He should be charged with criminal negligence and face prison time. Everyone is responsible for their own actions and its always possible to quit.

The account with the weak password was a former employee. It’s not on her/him that the account remained active and the admin password wasn't changed in the same process.

Rome wasn’t built in a day, not did it fall in a day. In a capitalist society you can gauge overall direction and success of the society but how well the market and private enterprise is doing, and well not merely in context of maximising shareholder value but as a fundamental part of the social fabric.

I've tried reading your second sentence several times but I'm unable to parse it. What exactly are you trying to say?

By* how well

Ah. I just totally disagree with you then. I don't think the market doing well at all implies that the "fabric of society" is also going well.

In Capitalist society, it is the prime indicator.

Because when markets aren’t doing well and you aren’t a socialist society, everything goes downhill. Look at the affordability crisis and its impact on quality of life in the west for a good example of it in practice.

If A implies B, that doesn't mean that B implies A. Having a bunch of nukes targeting every major city in the US would also make society not doing well, but that doesn't mean that society is doing well solely because there are no nuclear strikes here lately. My point is that I disagree that the market doing well necessarily implies that everything is fine.

Your point is entirely correct. Markets doing well alone is not enough, but at the same time, markets are a fundamental input to quality of life in capitalist societies. Both of these are true.

Seems to me that the implication is western us led society is collapsing like Rome did, but far faster, with years rather than generations.

"I dont understand why there is not massive reorganisations in systems when things go wrong"

Because massive reorganisations can easily lead to even more things going wrong. Also most people are lazy and phlegmatic by default.

A major problem we have is that computer programming is not engineering as people like to say. An engineering discipline VALUES redundancy and is always designing for safety. Programming likes to think of itself as math, and deliberately choses less redundancy for the sake of convenience and speed. The classical example is how operating systems are written with languages that allow an index to be out of bounds. We now have systems that are glued together using bubble gum pretending to be safe, when they fail in the most horrible way when one of the links break.

> I dont understand why there is not massive reorganisations in systems

This would just replace one insecure system with another.

It is time to recognise there's no such thing as a secure connected computer. And thanks to "AI" there's no such thing even as a significant defence lead over attackers.

They way I see it, there needs to be enough slack in an organization and the timelines for major products for people to not do the bare minimum. When management pushes goal X, and pushes hard, everything else starts to decay, including security. People need enough time to do the things they know they should do, but don’t feel they have the time for. At least this is where I’ve seen a lot of issues arise.

Yes I completely agree. In the local news there was focus on which company it was and that the password was 123456.

Of all the things that failed for that leak, we should focus the LEAST on the password being insecure, and the company whom had their account misused, and the most at the other end of the long line of failures.

Why was there no monitoring on a company suddenly looking up 600 people a minute, why was this only discovered when they were making the invoice?? And how was it even possible to have a password that unsafe, no two factor auth etc etc etc.

The older the system, the higher the chance it never got a proper security audit, and/or it was built with a lot of implied trust, like most old Internet standards are.

As for 2FA, it is a nice thing to have, but it comes at a significant support cost. People lose their token, people get annoyed by the friction, people can't figure out setup (especially older folks).

Oh no! Cost! Friction! Better just half-ass it then.

We're talking about a country with 6 million inhabitants, of which a large number is old and/or barely computer-literate and/or barely actually literate. Or doesn't possess a smartphone at all and only uses government services from a public computer in a library.

Government services have to work in all these scenarios, simply because that is a right of the citizens.

That's not how auditing works as I have observed it. Either your stuff is critical, or not. And when it is then everything which touches data sees an audit and no password policy incl. Shared and weak credentials is the first thing that would have been spotted. I would assume they buried some stuff to deep in a hierarchy and 3rd service partners that this company in the end got no proper audit.

Maybe I'm missing the point but isn't the parent comment asking with the admin accounts had no 2FA?

The security audit recommended changing the password to 234567, but management rejected it because it would require retraining staff

Im not in tech but still in corporate. Our store went through 6 GMs in 5 years. The problem is actually the corporate, not the new guy every 8 months who is being brought on to save the day. I think they're going to replace him again next year without addressing any of the issues on the ground.

The incompetence will continue as long as people can profit from it inconsequentially.

Our system now heavily reenforces lack of accountability to executives for what happens under their watch.

Investors don't lose money when these breaches happen.

This is why it won't change until those change.

Let's also include the system itself, that allowed the account to have a password that short and apparently didn't require 2FA.

When you have bad practice and process it can erode the security discipline of everyone working within the system. Until they commit brazen crap like reusing simple passwords everywhere.

In a country like Denmark it's very common for a higher up to resign over something like this.

sepuku would be more appropriate

> I mean you simply don't believe it until you witness it. Its just moral/leadership decay.

I think there's also a big part of the line that Jennifer Lawrence says in the satiric comedy "Don't look up":

"They are not even smart enough to be as evil as you're giving them credit for."

Just as the person who picks 123456 as a password doesn't see where the problem is, I think there are really quite a lot of people dumb enough all along the decision chain to think that firing the person who reported the problem did actually fix the problem.

They are really that dumb.

i find it is usually most useful to blame systems, not people. of course, it is usually _easiest_ to blame a person, rather than go through the hard work of redesigning the system. hence why:

> I dont understand why there is not massive reorganisations in systems when things go wrong. Its always , oh yeah so we fired the guy furthest down the ladder.

> I dont understand why there is not massive reorganisations in systems when things go wrong. Its always , oh yeah so we fired the guy furthest down the ladder.

Management should be tried for gross incompetence with some painful consequences severe enough to cause change in behavior. This is the only way.

Because everybody that wants to deflect blame off themselves pushes it down the organization, and lower levels of the organization have no real power to push that blame back up. That would require organized labor independent from company management.