Which is itself separate from the question of whether or not Mark Zuckerberg is being kind of a dick.
The article isn't bad and the reference is worth reading, but it's impossible to engage with this much material at the same time as though it's one conversation. It's poor intellectual discipline, I'm sorry.
What price is an airline ticket on the first day of a global pandemic? No one knows, and there isn't anything to provide a "procedural system" in order for it to model what our "intention" would be in those cases.
This seems obvious with a pandemic, but it is common place. Every day is a genuinely novel occurence, and counts only as a repetition of some superficial patterns in very very limited respects.
When this novelty becomes relevant to a problem can never, (obviously), be pre-stated because we don't yet know what that novelty is.
There is simply no substitute for being-in-the-world as animals are, that is: moving around, communicating, acting, reacting and participating. Growing, learning, reproducing, and problem-solving in direct causal contact with reality.
This isn't magic, but there is no "procedural model" of it; nor should we expect there to be. There aren't procedural models of almost any phenomenon.
What we can build a computer to do is "almost nothing", and we shouldnt expect the trivial problem space given by this to exhaust what we actually need machines to do.
You're stating that a large part of human behavior is reacting to the environment presented to them via sensory input/output mechanisms. What you have not stated is why a machine entity cannot do the same things?
Nothing, because any procedure that exists was necessarily defined by humans. If the procedure is implemented by a computer, at most that provides humans an excuse for choosing not to disregard an outcome it produces.
> Another thing is that Weizenbaum and the author seem to take it for granted that human decisionmaking is literally magic
I can't imagine where this idea is coming from. The thrust of the argument is not that human decisions can't be modeled, but that omitting the question of whether they should be modeled, and those models granted (what is necessarily the pretense of) authority to make decisions in human stead, evades the same sort of responsibility that a civil engineer has to ensure that his bridge design can withstand the mechanical loads that it will need to bear in use.
As for "distinguishing human and procedural decisionmaking" I would define it as basically "human" decisionmaking is when there is some procedure but the human will disregard procedure based on some model of the "intent." You have to trust the human for this to work. So there's a fourth question, which is under what circumstances you could trust a computer to correctly model your intent. But then there's a fifth question, which is "whose intent?" But whose intentions the process serves isn't really intrinsically tied to whether humans or computers are running the procedure.
Another thing is that Weizenbaum and the author seem to take it for granted that human decisionmaking is literally magic and a computer is incapable of that kind of magic. But it's unclear and I think it's an important thing to state outright.