I found it fascinating to learn that UK law explicitly does (but did not always) encode this presumption:
https://emptycity.substack.com/p/computer-says-guilty-an-int...
(This link is to the substack of David Allen Green, an excellent legal commentator in the UK.)
It's easy to see both sides of the decision to presume or not presume that machines work - we should like somehow to explain that on the one hand it's unlikely that without his intent Jim's iPhone sent Gemma a series of messages threatening to rape her, and so we shouldn't need to drag Apple engineers into court to convict Jim - but then on other hand it's far from certain that this bank database which says I made 590 cash withdrawals in sixteen minutes is correct, more likely a bug duplicated a single withdrawal I made, and we should require the bank to produce evidence otherwise if they insist I made these withdrawals.
I can imagine that smart lawyers motivated by some real world examples and assisted by independent software engineers could come up with a general formula for courts, something a bit larger than either the old rule or the new one, but still relatively pithy. Whether a government would ever make time to do that work is another question.
https://www.prospectmagazine.co.uk/ideas/law/64464/post-offi...
Fundamentally, the scandal is not merely a result of misplaced trust in an IT system. It exemplifies a series of injustices, overlooked for over 15 years by complacent politicians. Crucially, it illustrates the extent of deceit possible when bonuses for senior management and technical staff are at stake.
The UK legal system's appalling quality are starkly highlighted once again. Hundreds of postmasters faced wrongful convictions and prosecutions. Individuals were imprisoned for years, without any concrete evidence of their guilt. A single miscarriage of justice is concerning, but when there are over 250, it raises serious questions about the competence of the judiciary.
Astonishingly, more than 700 postmasters were prosecuted without any executive, manager, or statistician recognizing this glaring anomaly.
The longevity of the issue is also troubling. Even low-level Fujitsu employees, aware of the system's flaws, remained silent. Despite widespread media coverage and a string of wrongful convictions, leading even to suicides, not a single person within a large team exhibited any kind of moral responsibility or something that looked like a conscience. Their silence, just makes you believe again in conspiracy theories, and raises concerns about the integrity of modern IT systems and AI that we rely on.
The simple fact that somebody decided to attest to the absence on bugs on an IT system...An achievement unheard of in the history of software development, and the fact they decided to reiterate that under oath, at a court of law, should be enough for their names to be published and banished for ever, from any profession related to technology.
They were fully aware of the problems and deliberately not acknowledging them.
e.g. the ongoing inquiry has just been shown a document showing the Post Office indicated that it would accept a guilty plea on a less serious charge provided the defendant agreed to accept that there was “nothing wrong with Horizon”.
As far as I am concerned, justice will not have been done until all of their bonuses are clawed back, and at least one of the execs has been smeared on tracks by the tube.
I was thinking about this last night. I knew there where about 1000 McDonald's in the UK, Googling it there are about 11,500 post offices. At some point you'd have thought someone would have noticed it was a bit excessive.
>
> The simple fact that somebody decided to attest to the absence on bugs on an IT system...An achievement unheard of in the history of software development, and the fact they decided to reiterate that under oath, at a court of law, should be enough for their names to be published and banished for ever, from any profession related to technology.
Completely agreed. This is why ethics in our profession are important, and yet all too often dismissed.
That's not true.
>Richard Roll is the real life Fujitsu whisteblower behind the character in Mr Bates vs The Post Office who lifted the lid on the faulty computer system.
I don't think this is a real thing. The individual system was vouched for by senior people. That's why it wasn't questioned any further.
"In 1997 the Law Commission published a paper which went into some detail about the use of mechanical and computer evidence in court. It seemed a little too fixated with the effective workings of speedometers, traffic lights and breathalysing devices called ‘Intoximeters.’ It concluded that the present law is ‘unsatisfactory’ because of the necessity for prosecutors to ‘prove that the computer is reliable.’"[0]
The amended law changed the burden of proof from the prosecution proving the system functioned correctly to the defence proving it didn't, without access to the systems being used to prosecute them.
[0]: The Great Post Office Scandal, Nick Wallis
It is mind boggling that someone thought it is OK to put this into law. What happened to the idea of innocent until proven guilty?
(not that it’s fundamentally just with speeding tickets either really! but it's also kind of understandable, you're the 27th person the judge has heard today trying to weasel out of a speeding ticket with the same set of "but the machine could have been wrong!" excuses. And frankly the brits seem to be a lot more "pragmatic" about individual right vs societal ones... the political class there seems to respect the people there exceptionally little even by political-class standards lol)
he breached his bail conditions and was rightly jailed for it
Judges in the UK have been so impartial they publicly made statements of him being a narcissist, not even pretending to even appear impartial. Nils Melzer, the United Nations special rapporteur on Torture and Other Cruel, Inhuman or Degrading Treatment or Punishment, concluded after visiting Assange in prison is treatment was nothing more than torture...
Yeah totally normal...for the UK legal system...
it turns out there's consequences for jumping bail
The fact that the Fujitsu employees had complete and un-audited access to all the remote terminals should have made the computer testimony unreliable, but of course the post office lied about that and denied that anyone had remote access.
In the case of the sub-postmasters the Post Office, as far as I'm aware, never proved where these stolen sums supposedly went. The computer evidence was thought terminating and was the only thing (except false confessions under duress) used to secure these convictions, rather than proper investigative work.
It's crazy auditors didn't spot discrepancies especially with the high base rate of reported frauds and errors, but if finding the receipts for stolen money was the threshold for prosecution any remotely competent thief would be in the clear
If there is absolutely nothing to corroborate the money going missing, no evidence presented for where or how the money was stolen then there shouldn't be enough to send someone to prison.
The amounts of money involved may be small to business owners in other domains but for SPMs many were almost bankrupted by trying to replace the sums out of their own earnings as you say. This wouldn't have been a rounding error to their lifestyle, it should have been provable to any halfway decent investigator. And if not? Then they get away with it and it's the price we pay not to live in tyranny.
Well, yeah. A defendant should be able to argue that because that can (and did) happen and it's bad to routinely wrongly convict people.
I thought when you called it a "tricky one" you were expressing that it might be a bad thing if it were difficult to convict someone based primarily on audit logs.
But if you don't want people to be wrongly convicted, then surely that's a good thing, right? As we know, there's no guarantee a particular audit log is correct.
Think of it like a diagnostic test, like covid tests. That sort of test has 2 measures, not one (anyone who just says "Test X is 95% accurate!" is selling you something) - specificity and sensitivity. Sensitivity is the percentage of true positives it generates out of all positives, and specificity is the percentage of true negatives it generates out of all negatives.
I don't want people to be wrongly convicted, no, so I want legal tests to have a very high specificity. But I could do that easily, by just throwing every case out as not guilty. The hard bit is raising sensitivity at the same time. You can't just say "if you don't want people to be wrongly convicted", because that justifies far too many things.
> As we know, there's no guarantee a particular audit log is correct.
There's no guarantee anything is correct. Three witnesses could have colluded and someone might go to jail for it, but unless there's a reason to think they colluded, we don't assume that. That's the problem I'm talking about: how do we get a feel for software systems without assuming like the Post Office that they work, or like you that they don't work?
Well, how the Post Office commissioned the Horizon system is something that the inquiry doesn't seem to have got to yet.
From what I've seen of the inquiry (not much), Post Office managers seem to have been desperate to protect the reputation of the Horizon system, and still are. But the miscarriages of justice have now been exposed, and the costs of putting them right are now unavoidable; so the reputation of Horizon can't now be just a matter of subpostmasters.
My guess is that defects in Horizon, and damage to Horizon's reputation, will turn out to affect much more than the wronged subpostmasters. I suspect that some very large commercial contracts will turn out to be affected, and paying compensation to the subpostmasters will turn out to be a tiny component of the costs of this catastrophe. The Post Office is already warning that it may not be able to pay the compensation, and that the government will have to step up.
The Post Office is a weird mess of subsidiaries and holding companies, and has contracts with all sorts of 3rd parties. I anticipate that we are going to witness a slooow explosion of the Post Office, over like ten years. It's ludicrous, of course, that a private company should be able to be complainant, investigator and prosecutor as well as being sole custodian of the evidence (Horizon logs).
It's also ludicrous that the Post Office's lawyers, Cartwright King, should describe requests for disclosure as a "fishing expedition". Defendants shouldn't have to request disclosure of evidence that might help the defence; that is an obligation that falls on the prosecution as a matter of statute law.
I'm waiting for the inquiry to find out how the Horizon system was commissioned, and how it was tested; and who was in charge of commissioning.
It was only subsequently that that believe changed to outright lying to cover their own backs.
As can be seen by following the story, this wasn't the case. The only evidence provided in criminal cases was essentially a witness statement saying that the system was working correctly and that was the end of it.
I'm not saying it wasn't. You're assuming a blind faith in all IT is the cause of this witness' belief. I'm disagreeing with you there.
> it was also enabled by a belief in the reliability and credibility of IT systems
The obvious mistake most likely made here is these senior people where not asked , " exactly how they formed this opinion." and what specific evidence they had.
I would have said , if they could NOT produce the evidence , that verified their statement, their is no way they could make the statement as accurate about the system accuracy.
The people must not have had experience in software development and testing.
Fujitsu was a contractor; the contract was a PPI (Public-Private Investment) contract. The deal was that Fujitsu developed the system at their own cost, and ran it in exchange for a cut of each transaction.
I know something about this work; back in the 1980s, I worked on a bid for what was then called Post Office Counter-mechanization. This was probably the third attempt to automate Post Office counters; previous attempts had failed. FWIW, we lost the bid; I think it was won by ICL, the forerunner of Fujitsu. That third attempt failed, of course.
To work on this bid, I was loacked away in an office on my own for 6 months. The requirements were delivered as two metres of lever-arch files, full of paper forms; each form described an input screen, or a report, or a processing operation, or a storage requirement. 90% of these forms were blank, apart from the name of the operation. It was impossible to do a serious job when so little of the required data was supplied. And we weren't allowed to talk to the Post Office about the requirements; that would have risked the entire tender being overturned in court.
If the Fujitsu PPI contract was handled in the same way by the Post Office, and I assume it was, then it would have been impossible for Fujitsu to deliver a working system. But now we see Alex Chalk, the (current) minister in charge of the Post Office, suggesting that the compensation money should be extracted from Fujitsu. It may turn out that Fujitsu have some hard questions to answer; but for now, it looks as if this mess falls entirely on the shoulders of incompetent Post Office managers.
They knew the software had many bugs but decided to use that to bully their employees and steal money from them.
I don't really see how this argument fits. In my understanding the amounts in question were in the range thousands to tens of thousands. It seems implausible that those sums of money, while life changing for the individuals involved, would ever move the needle at all for any of the companies involved. In fact the only people who made money out of it are likely the lawyers.
Presumably those people potentially faced contempt of court charges if they continued to blame the system.
In some industries, like aviation (at least in EU) it appears to be more understood, but in general it doesn't.
I don't think anyone was trying to do the wrong thing but rather timelines and budgets were tight.
> Imagine joining an engineering team. You’re excited and full of ideas, probably just out of school and a world of clean, beautiful designs, awe-inspiring in their aesthetic unity of purpose, economy, and strength. You start by meeting Mary, project leader for a bridge in a major metropolitan area. Mary introduces you to Fred, after you get through the fifteen security checks installed by Dave because Dave had his sweater stolen off his desk once and Never Again. Fred only works with wood, so you ask why he’s involved because this bridge is supposed to allow rush-hour traffic full of cars full of mortal humans to cross a 200-foot drop over rapids. Don’t worry, says Mary, Fred’s going to handle the walkways. What walkways? Well Fred made a good case for walkways and they’re going to add to the bridge’s appeal. Of course, they’ll have to be built without railings, because there’s a strict no railings rule enforced by Phil, who’s not an engineer. Nobody’s sure what Phil does, but it’s definitely full of synergy and has to do with upper management, whom none of the engineers want to deal with so they just let Phil do what he wants. Sara, meanwhile, has found several hemorrhaging-edge paving techniques, and worked them all into the bridge design, so you’ll have to build around each one as the bridge progresses, since each one means different underlying support and safety concerns. Tom and Harry have been working together for years, but have an ongoing feud over whether to use metric or imperial measurements, and it’s become a case of “whoever got to that part of the design first.” This has been such a headache for the people actually screwing things together, they’ve given up and just forced, hammered, or welded their way through the day with whatever parts were handy. Also, the bridge was designed as a suspension bridge, but nobody actually knew how to build a suspension bridge, so they got halfway through it and then just added extra support columns to keep the thing standing, but they left the suspension cables because they’re still sort of holding up parts of the bridge. Nobody knows which parts, but everybody’s pretty sure they’re important parts. After the introductions are made, you are invited to come up with some new ideas, but you don’t have any because you’re a propulsion engineer and don’t know anything about bridges.
> Would you drive across this bridge? No. If it somehow got built, everybody involved would be executed. Yet some version of this dynamic wrote every single program you have ever used, banking software, websites, and a ubiquitously used program that was supposed to protect information on the internet but didn’t.
> You can’t restart the internet. Trillions of dollars depend on a rickety cobweb of unofficial agreements and “good enough for now” code with comments like “TODO: FIX THIS IT’S A REALLY DANGEROUS HACK BUT I DON’T KNOW WHAT’S WRONG” that were written ten years ago.
It's not all programs, but the ones that last (you know, the ones people actually use) get more and more like this over time, no matter how they start.
It's out-of-date though... allow me to fix it:
> ...with comments like “TODO: FIX THIS IT’S A REALLY DANGEROUS HACK BUT I DON’T KNOW WHAT’S WRONG” that were written *twenty* years ago.
(It was written in 2014, after all.)
- Weeks, months, or years of being incomplete and full of bugs. Gaps are filled and bugs fixed continuously, but new bugs are being introduced all the time.
- At the end -- where the end is a predetermined date and not based on the state of the software -- the software is deemed good enough and put into production
So most software is just barely working, at best, and full of gaps, issues, and bugs, both known and unknown.
This scandal is a case of egregious lack of leadership, both at the post office and at Fujitsu.
But it was also enabled by a belief in the reliability and credibility of IT systems, which is profoundly undeserved and inaccurate.