I think maybe you have to think about the total economic impact for the article and frustration to make more sense. Fuel is a very fundamental part of our economy and almost always goes toward producing much more value than the value of the fuel itself. Add to that, shortages can and do contribute to irreversible losses like loss of life. If every gas distributor shut down because ransomware hit them all, you could completely grind the country to a halt.

It's not too far fetched either to hope that their customers would be willing to self-report. Sure you might get some people who'd take advantage of the situation, but it's not a the total loss scenario of giving away your product for free that you're talking about. Add in the opportunity cost and shutting down delivery if you have alternatives that may not be 100% accurate billing is probably a net loss for the company, though it's impossible to say.

> I'm still trying to find out what actually happened in Israel but god forbid a news source or commentator actually lay out the details of a complicated situation... (if you're not getting everybody pissed off about X, then you're not doing your job as a reporter?)

It's a bit ironic to take a morals-over-profit stance on reporting right after having taken the profit-over-morals stance of gas pipelines.

> It's not too far fetched either to hope that their customers would be willing to self-report. Sure you might get some people who'd take advantage of the situation

Would you bet millions of dollars on the goodness of the typical person?

> Would you bet millions of dollars on the goodness of the typical person?

This is about business to business, not business to consumer. I suggest to talk to a financial/accounting person in a big company. Often enough there are mistakes in payments that are only noticed by the other business. This despite all the various layers that should've prevented the entire problem.

I don't mean small mistakes either, I mean where someone wanted to enter something like 1234.56 (two decimals), but it ended up as 1234560.00. Basically a mixup between the decimal and thousand separator; such mistakes happen way more often than you'd think. I've heard various stories of a vendor notifying the company that they've been overpaid, e.g. the 1000 times what it should've been, but also various other kinds of mistakes.

Some companies have been working together for various decades. Employees might've switched between the companies, people know each other. It really isn't uncommon to have quite a bit of trust between companies. Obviously, this does depend on the country. In some parts of the world there's more trust than other parts.

That's how credit card processing operated worldwide for many decades. That's also how hawala works. And Wikipedia, if you want a non-monetary example.

They are all literally trusting in the goodness of the typical person. And then doing some careful analysis and planning. And fraud detection for the rare misdeeds.

But a credit card still keeps track of how much you spend - it doesn't rely on you honestly reporting how much you spent over the month...

Oh, I think they are talking about the earlier period where you just did a carbon copy of the card, got it signed, and if the card was a fake there was certainly no way to tell at the time. It would be the equivalent of a bad check, talking the days before swipe machines.

I guess, but discounting active fraud, there is still a paper trail of how much you spent; The bank will still have said "We think you owe us $100, here is our reasoning based on things you signed" never "you spent some amount of money we're not sure of - please give it to us" or "all of our customers spent $1million, so let us have your share, whatever you think that may be" as would be the case here.

Yes, the system would fall apart if you disputed every single transaction and made them supply irrefutable evidence you did in fact authorise the transaction - but it doesn't revolve entirely on the trust that people will keep their own account of what they think they spent.

Sure, but the store that accepted the invalid credit card has traditionally been on the hook. The store is the one being targeted even if the person whose credit card was stolen can get off with disputing the charge.

And before everything was networked, there was no way for a store to check with the credit card company that the card wasn't reported stolen. That's the trusting people part, it's the store that is trusting people without the ability to even verify that the card hasn't been reported stolen.

Yes, there is a paper trail, but the situation I'm talking about is analogous to a bad check or a counterfeit bill -- there's no recourse if the bad actor can't be found and the store is on the hook.

Ah I understand now - from the shop’s point of view yes there is risk in this.

I am just about old enough to remember this time - it’s funny to think that security hinged on checking if the signature looked ‘right’. For cheques I remember here at least for yours to actually be accepted anywhere you needed a ‘cheque guarantee card’ which was basically just a debit card but with a fancy hologram - so a certificate of your ‘goodness’ - up to a certain limit per cheque. The shop would copy the details and this would supposedly guarantee they’ll be paid.

I was going to say that shops could have asked for ID - but at this time driving licences didn’t have a picture and weren’t cards - so no one carried them anyway. There was probably a lot of ‘subjective’ acceptance around the place (‘does this person look trustworthy, would they really be given a credit card or did they steal it?’) come to think of it…

Yes, I didn't want it to veer off topic, but I think it is extremely obvious that such standards involved implicit or explicit biases, as well as real recognition of individual customers' actual past behavior (like if a gas station has taken advantage of them in the past).

I assume the companies then just passed the risk onto the consumer (or ate it to grow their user base), I find it hard to believe they just continued operating on the honor system.

Would you bet millions of dollars on payment system with zero redundancy? Also with that kind of money involved I wouldn't be surprised if you could could get a court case running and force them to report how much they took.

I mean that’s how lots of business works? Is that not the whole basis of invoicing? I must be misunderstanding

An invoice would say “you owe $x for y gallons of gas” not “hey pay us some money for that gas you got”

True, but still: a lot of (big) businesses are still quite unhappy if they didn't pay what they should've paid. It also creates an uncertainty for them, there's often various years where the money can be requested back. Plus as far as I know, if you notice an error there's various financial rules you need to follow. You cannot just treat it as "cost saved". As a result, it is way easier to notify the other party and get a mistake fixed.

I think there's a way they could count the number of all the trucks without a computer.

When all parties are losing a bunch of money on acount of the shutdown, the rational choice would be to accept some lesser risk than these known losses in order to get things working to some degree, if that were technically feasible.

It shouldn't even be up to them to make a rational choice. It should be required via federal regulation on utilities.

No, I would create legislation that mandates that the pipeline must keep running in situations like this at the company's expense

> Would you bet millions of dollars on the goodness of the typical person?

Somewhere in that question is an answer to the Fermi paradox!

>It's not too far fetched either to hope that their customers would be willing to self-report.

Spoken like someone who has never done business.

Spoken like someone who would let critical public infrastructure get privatized by a corporate monopoly.

Somewhat of a moot point when said infrastructure is already private. You can use the misaligned incentives to argue against private ownership of infrastructure, but criticizing a business for doing business is barking up the wrong tree.

It's not a moot point if we can create regulation or government takeover of said infrastructure. Conversations like this are the first step towards creating those kinds of regulations. It's only moot if you make it moot. When a private business completely controls a public utility and fails to act in the public's interest it is time to have this conversation because they have their hands all over something that should be in the hands of the public.

My point is that any business would have done the same. Expecting one to not look out for its own bottom line and rely on the kind heartedness of its customers is misguided. The problem is with the game, not the player.

I wonder if there is a capitalistic approach so solving this problem (making incentives align between the needs of the fuel distributor and that of the economy in the case of technical failure)?

The only approach I see is perhaps to have written into contracts with all buyers some kind of compensation for the pipeline being unavailable per hour.

Then colonial will prefer to have the pipeline turned on handing out free fuel because it's cheaper than the compensation for turning off.

If only the world economy ran on tears and sympathy instead of recorded transactions. I should pray that all my utility and food providers function without charging me money... until they go bankrupt and I am left with neither.

You'd probably be pretty pissed if your electricity provider suffered a ransomware attack on its billing system and then decided to stop your heating completely on its own accord because it can't know if you're paying your fair share.

They do that anyways here even without ransomware attacks. There were all kinds of articles about it on the front page of HN.

When?

Texas recently shut down their power grid when it got cold out because it was cheaper than weatherproofing or suffering the regulations to interconnect with other neighbouring states.

That's not in any way similar. The Colonial Pipeline was not shut down because they ran out of petroleum to distribute.

Grandparent commenter with "Austin" in his name was saying his power gets shut off even w/out ransomware. Parent commenter asked "when?" and I answered the question.

I'm not taking a position on the similarity or dissimilarity between shutting down a service because you can't bill accurately and shutting down a service because you refuse to interoperate with other providers.

All I'm saying is Texas customers did get their electricity cut off recently.

The Texas power grid didn’t run out of fuel either. It was known there needed to be controlled outages a couple weeks in advance. What wasn’t known was the scale or duration of the problem.

At any rate you are just looking for something to cry about. You asked for a hypothetical counter example not expecting to get one and now are splitting hairs to qualify big tears.

>The Texas power grid didn’t run out of fuel either.

"As of Wednesday morning, when the power outages were at their most severe, the cold had snuffed out about 46 gigawatts, or about 40 percent, of power-generation capacity in the state."

Sounds a lot like the grid did in fact run out of "fuel" (as in the product that they deliver).

>You asked for a hypothetical counter example

Of an electricity provider cutting power of its own accord because their billing service was not functioning, not because the electricity production in Texas dropped due to freezing while demand shot up.

There are some restrictions on when the utilities can shut off your power, water, electricity, etc. Whether gasoline should be regulated like a utility is an interesting question. Given how dependent on cars most Americans are, seems like a reasonable conversation to have.

Frankly, you seem to lack knowledge here. Your example of a grocery store is B2C with no knowledge of the customers. The pipeline is B2B with legal contracts in place and use invoices.

You can order dozens of types of oil through the pipeline. With travel distances of thousands of miles, many destinations, and the need for product segregation, the pipeline knows where all the oil is going.

If they couldn’t track segregation and destinations in the pipeline, that is a good reason to shut down. If they were just going to be delayed in sending out invoices or had to input some items into the system manually after recovery, they most definitely should not have shut down.

> The pipeline is B2B with legal contracts in place and use invoices.

What if billing is tied to metered usage and it was impossible to meter? Or what if the billing system is needed to shut off access when customer has reached a quota? The article is scant on details but "legal contracts and invoices" are not going to save a company legal fees with when multiple customers dispute their bills.

>If they couldn’t track segregation and destinations in the pipeline, that is a good reason to shut down.

Wouldnt that then conceded the pipeline was not fine?

>If they were just going to be delayed in sending out invoices or had to input some items into the system manually after recovery, they most definitely should not have shut down.

I am willing to bet a signifigant number of "customers" (businesses in this case with teams of lawyers) would challenge the vailidity of invoices based on estimates and not actual usage. Theres no free lunch, the pipeline owners understand this and decided its in the best interest for their company to stop deliveries until the system that manages their order tracking is back online. Thats totally reasonable.

The US government (and states for that matter) made 0 guarantees of coverage of any thing. And even if they did, its not like the government does much to make anything a straightforward process.

It's understandable why they would shut it down temporarily if they could no longer accurately bill customers for usage.

At some point, however, it starts impacting the economy and there would have to measures in place to get the fuel going regardless of whether the corporate bean-counters (and their malware infested computers) were fully "ready". I don't know what that point in time should be. But the fact that there were news stories of imbeciles hoarding gasoline and emptying fully loaded gas stations within a couple of hours certainly makes it urgent to get the pipeline running sooner. Can't we all just get along?

>Yes, in fact, if a store's computers go down they don't just let customers walk out the door with products and hope they get paid eventually?

They better do that if they get the permission from the gov't to operate the only grocery store in the area, and people weren't able to get food otherwise.

Okay, so establish some sort of "crisis service fund" that pays out when payment processing breaks through no fault of your own. This is something the state can do at any time. Honestly, allowing a monopoly without planning something like this just seems careless.

Providing service without hope of ever being paid is stupid. The market is pretty good at selecting against stupidity.

I lived both in a high-trust society and in a low-trust society. That you seem to have experienced only low-trust society may impact you worldview, but take my word for it, things are different elsewhere ;-)

I really like this comment. It is an indictment of our society if we feel like we can't trust anyone around us. I have people around me with whom I'd trust my life; it is much better to surround ourselves with those people. And there are lots of examples of societies historically and now where things operated very largely on trust.

Don't be too harsh. The experience of living in a high-trust society is becoming more and more rare for future generations, due to certain policy decisions. They will just assume things being broken is the natural order.

> Okay, so establish some sort of "crisis service fund" that pays out when payment processing breaks through no fault of your own.

One company which suffered a cyber attack solved this partly by transferring a rough estimate of what a vendor should've been paid as a bank transfer. This up front. They did similar things for smaller vendors (smaller vendors might have bigger issues not getting paid on time). Not for everyone due to the amount of work involved. They got agreements in place to sort out the exact amounts afterwards. This required a huge trust and at the same time it created a great amount of goodwill from those vendors. Meaning, these vendors took over work on behalf of the company.

Afterwards the exact costs can be figured out. What is important that business continues as much as possible, if you completely stop your business you'll also not have any revenue you could maybe charge later.

I think the pipeline action on first hand is a rather stupid action. Loads of business is pretty predictable, plus often you already had the insight on the orders for the coming period. The cost of some mishaps and the economic damage to the country should be considered as well. It seems a case of "penny wise, pound foolish".

Regarding "payment processing" breakdowns: there are various things in place in NL for that, though most don't know. Simply stated: you fill out a paper form, sign it, done.

Sounds like solving the problem from the wrong end. The payment infrastructure is the service provider's responsibility, not the state's.

Instead, set up strict SLAs with harsh fines; make it more expensive to shut down than to provide free service.

There are ways of collecting payments that don’t involve contactless card terminals.

The other thing is when you have hackers in your network, shutting down the things that could be physically dangerous if they access it, until you know what you are facing and you identified the compromised machines, doesn't seem unreasonable to me.

That depends on what the alternative is. If it involves clueless people filling up garbage bags with gasoline and storing them in their kitchens perhaps letting the pipeline continue to run would have been safer.

it's also sounding kind of like the simplification an IT clueless manager might make after too much coffee and not enough sleep.

it whats likely happened was that the hardened industrial control network was pretty much fine meaning that engineers inside of the pumping facilities could manually direct/redirect flow with all the required safeguards still in place, but that they had no plan for where to direct the flow as all information about who needed/ordered what sat in the unhardened corporate network.

This is kind of similar to what happens to warehouses and "just in time" factory lines when the order printer/queue ceases up, or is no longer fed data from the back end system, and while one might expect that the operators of critical infrastructure have a plan for minimal services when/if the unhardened corporate network fails but that plan(even if it existed) likely sat on the same unsecured and now encrypted network.

> Yes, in fact, if a store's computers go down they don't just let customers walk out the door with products and hope they get paid eventually?

It's one thing with a supermarket and anonymous masses of customers, but an oil pipeline where each customer is supposed to have meters running on their side as well (as double verification to protect against technical issues)? If you can't trust your known (!) customers enough to not rip you off, something is very much off.

And even in that case: the government could have told the pipeline owners "we'll pay off any differences after settling, now go and get that pipeline up and running again".

I was in a fast-food restaurant when their credit card processor went down. They continued to take orders and if the customer didn't have cash the food was free. The cost of poor service and losing repeat customers was greater than the reduction in profit.

Now, this was a semi-independent franchise owned by a local family. Across the street, the chain big-box store had the same thing happen and they put a sign on the door saying "cash only".

> if a store's computers go down they don't just let customers walk out the door with products and hope they get paid eventually?

This is exactly what we expect them to do in a crisis situation. This is also how Hollywood portrays ethical behavior. The store owner in Jericho (2006 TV series) is the first to come to mind, but there should be plenty of other examples.

And good deeds come back to us.

That sort of thinking brought down the USSR, nothing was paid for and the commie elite sold shit for US $ and wasted it. In time this system would have no oil as well as no $$ and all the pipes would be trashed...

The difference is that the Colonial Pipeline is seen as an important part of the US Critical National Infrastucture - without oil large parts of the economy and security system breakdown, people can die. It's protected by Homeland Security and the FBI too a much greater extent than your average store.

Ok, so the government steps in and says the pipeline operates without a billing system... now what?

Pipes are useless without something to flow through them, and no oil producer is going to fill the pipe without a system to get paid.

The story was that hackers had closed the pipeline (or the owners had closed it because its systems become compromised).

If the pipeline is seen as a piece of critical infrastructure [1] - where its closure can damage other parts of the US - then if Colonial didn't tell the FBI, CISA, the Department of Transport (or whoever is in charge of that part of infrastructure) that the reason they were closing it was because they couldn't bill their customers rather than it being a safety issue, I imagine they are in trouble.

They were lucky in that it was only down for a short period, perhaps its OK - if it had been down long enough so ambulances stopped running and food and goods weren't delivered - then I'm sure the government would have told them to restart the pipeline and billing would have been sorted out some other way.

[1] https://en.wikipedia.org/wiki/Critical_infrastructure_protec...

This wasn't even a shitty take -- most of the developed world has government-owned utilities so the thought that the entire east coast gas infrastructure could go down just because some private company had their billing infrastructure locked up is not only laughable, it's morally outrageous. This situation is just begging for federal regulation, and that's the conversation we should be having -- should laws be passed requiring companies like Colonial to keep pumping gas (on their on dime) even when their billing infrastructure goes down? I think yes and I think most of the general public would as well.

Instead of having that discussion, we have no discussion because people like you have decided to flag this issue. This is the problem with HN.

I agree with you that the company can't make a decision to simply ship fuel for free. This is where the government should step in. You can't have this kind of wide-scale economic disruption occurring. Government could mandate that they resume shipments, have them bill manually as best they can, and maybe use tax dollars can plug any holes after the fact (and if necessary).

They should also be fined for letting their billing system get compromised.

I think the point they’re really trying to make is that gas should be treated like a utility - you shouldn’t provide it solely based on profitability for a corporation. But you are right that the way things are now, it’s quite clear why they shut down, and that outrage at this particular company seems to really be outrage at the system which it operates in.

>Of course they shut down if they couldn't bill?

I know it's a commercial operation but there's strategic reasons to keep it up. The attention it got from the US is evidence of that. Plus, they could have just made an educated guess on billing and resolved it later. Even the US could have made cash flow happen if that was an issue. And anyone who tried to take advantage of that would likely get the US's attention. Not worth it long-term.

But all this was overshadowed by the risk of ransomware possibly attacking the industrial systems which is really why it was shut down.

Regardless, the moral of the story is this:

To hack and shut down infrastructure does *not* require penetrating the actual infrastructure. Instead there are (likely far more vulnerable) "support systems" that if compromised will have the same effect. Why pick a hardened target when a soft one will do?

It's like arresting Al Capone. He didn't go down for all the violence, etc., instead it was tax evasion.

Isn't the argument that it should be a utility and not a purely for-profit business? Would you expect your local water company to shut off supply if they have a similar bookkeeping problem?

People, and particularly software people I've noticed, seem to chronically underestimate the magnitude of B2B where you have a potential number of different vendors to settle with in the 6 digits, that changes over time according to vendor internal reorg, closings, openings, relocations, changes of payment and financial service outsourcing arrangement, etc.

That doesn't at all come for free. Entire departments and businesses revolve around making that complexity disappear. Gettting locked out of your infrastructure to do that is essentially paralysis of your business period. It isn't a case of "Just put some meters on the tap of the pipe" .

You have to be able to read them, you have to know to read them and when, you've got to know where the invoice goes, what terms were negotiated, etc. If you don't have access to that dataset, the only physical way for the human network to recreate that web is by people picking up phones and blowing up and slamming your point of contact which runs into the issue of having enough people on your end to handle things in the meantime, and your customers actually knowing who to call, and javing planned a procedure for a massive degradation in your info system, even if it means going back to pen and paper.

The author here is misrepresenting his sources to spin a specific agenda point, in order to construct a false narrative. It's as fake as fake news gets.

From the author's own sources...

> Colonial’s corporate IT network and the process control network are connected and exchange information about how much fuel each supplier or distributor receives in order to bill them for it.

> ...they could change [data about the] flow rates, they could modify the data

> Although infecting Colonial Pipeline’s process control network would be disruptive, it isn’t the only concern. Colonial’s control system also connects to the control systems at tank farms that feed fuel into Colonial’s pipeline.... An attacker can potentially pass through Colonial’s control systems into the control systems of these farms.

I really hate advocacy journalism. It's such a fundamentally dishonest medium.