Middleclass
In the analogy, the specification is the information provided to government officials and binary code is planned governmental actions.
If both are known and the specification process for governmental action is proven it doesn't matter what biases the officials exhibit in the decision process - then the governmental action is proven-good according to the specs (e.g. system of laws) and the information provided (in which it would be relatively easy for citizens to have a say, e.g. "you MAY NOT ignore facts X, Y and Z").
Your point stands, nevertheless.
A black box might work IF it can be ensured that all data considered will be bundled with the decision. But how can such a thing ever be ensured?
We'll soon have source-included but secret-compiler software as a technical possibility, by means of zero-knowledge computational proofs.
It seems a bit much to ask politics to follow that state-of-the-art, which means that closed-doors talks* should be outlawed if the EU wants to have leg to stand on v. rising anti-EU sentiments.
*. indefinitely-closed-doors talks. A mandatory post-mortem including release of the talks themselves after a set period of time would go a long way.
What I meant to say was this:
"The lumping together of outcomes in roughly two boxes, while also discarding information about the specific actions that lead to the outcomes."
Once this has been done, there is no longer a line back from the outcomes ("buying food") to the actions that led to it ("receiving a salary"). Or "finding a job for some or other purpose", which leads to "working for company X", and then "receiving salary from company X", and then assuming part of the responsibility for the actions of "company X" by spending the money earned there.
I know almost no one who considers where their money comes from, and then actually implements steps to make their spending habits congruent with that.
Ethical use of money does not just involve spending it at the right places, it also involves earning it in a way congruent with where you spend it, e.g. keeping track of where the money originates from. Companies do not create money and everyone who works for them knows this.
For instance, if I receive money from place A which is involved in studying agriculture, if I then spend it in a place which is agriculturally underdeveloped, anything I accidentally reveal about agriculture to the place where I spend my money is well placed.
In the same vein, if I'm part of a criminal underworld involved in child trafficking and prostitution, I should not be spending it on schools, even if it were possible to keep the two worlds completely isolated in all my other actions. Thinking such a total isolation is implementable by hand without actually knowing you can do it and how, is highly unwise and a sure sign of an overgrown ego.
Taking money from one place and moving it to another just creates an undeniable and indestructible trace from place A to place B. If you are not a perfect filter to things which are at place A and should never be at place B, you are responsible for any and all harm caused, through willful ignorance of the fact that companies make money in cooperation with the society they function in.
To give you a simpler example, if I'm developing software that pushes some kind of social frontier, it makes sense for me to import my food from the most socially backwards countries, so that I don't accidentally mistake the actual problem space for "just my backyard". Because then I literally owe my food to the support of a country which has bigger social problems than I do personally.
Ideally this makes the software I develop less complex, I get a wider picture of the problem which makes it harder to even consider complicating the solution, and it helps aim my attention more precisely in ways which are known-good.
However, if I only were to consider the outcomes of my actions, such as great software or great tasting food, I may be effective at producing sharp weapons but not so much at producing tools for peaceful ends.
Back in 2017 when I had a testing automation job, I wrote a test automation system using Node, Selenium WebDriver, Cucumber and Vagrant.
It worked well once I managed to set up a Vagrant box that would take Cucumber tests from a local directory and keep a Node, Selenium WebDriver and Cucumber install cached, but WebDriver never really stopped crashing unpredictably.
I had to implement very coarse retry logic. Tests would take way too much time just because each run a few of the tests would keep crashing for a minute or more, until they finally succeeded.
I parameterized the Vagrant box so that testers could run subsets of tests by running the "test" command with parameters, not because we had that many tests (just about a 100) but because they were so slow.
It wasn't even that complicated of a SPA, and the backend engineers even added nice classes and ID's to elements that were to be tested.
The binding of Cucumber to JS to WebDriver was flawless, adding new testing functions (e.g. "do something with some particular type of list of items"), it was just that the browser component kept crashing all. the. time.
I longed for a deterministic means of automating the browser then, preferably by hooking right into the browser code and integrating with it, so I would know why the thing kept crashing. That hasn't happened yet.
It uses YAML too, but always felt much more cleanly composed, and smoothly integrates with scripting languages.
I've never truly broken into orchestration and am still salty about that so please, take my post with a grain of salt.
(Full disclosure: I have literally no professional affiliations whatsoever.)
What bothers me about some utilitarian perspectives is the lumping outcomes together of outcomes, based on vague definitions of benefit and harm, instead of keeping the effects strongly tied to their causes.
If you take that approach joining a company, you better know how much of the company money passes through you, and you better spend it very ethically and keep track of the development of the ethics of the company.
"Knowing a tree by its fruits" and all.
AFAICT it's the philosophical blind spot that got us parallel construction (an illegal US police practice), manufacturing evidence based on desired convictions, to reach a desired monetary, political or personal end.
There's also a New Age concept for it, it's called "spiritual bypass". "If only I'm spiritual enough, I can do whatever I want", or "if the Lord wanted my garden to bloom he'd water it, I did my part praying".
In psychological terms it's called "rationalization". I don't think it's an avoidable abuse as long as you don't do your own bookkeeping.