There are fixed costs in the development of the system but there are running costs (owning or renting the servers, improving the ML and the system at all points, gathering more data, curating, maintenance of the data, etc). I don't think anyone reasonably believes that we can just build a box of truth and that nobody need be involved in that project ongoing afterwards (even just to ensure future browser compatibility, security upgrades, system outages, bias in data gathering, etc).
What I would be interested in seeing is what the very same money put towards such a project would do for a system that desperately needs much more influx of money towards structural pillars like mental health, housing, legal counsel, etc.
Since there's seemingly no A/B testing (with what I mentioned), no double-blind testing (which would test for the impact on the judges), or any form of comparative testing really, it's pretty hard to conclude a lot of things about this project. We surely wouldn't go too quickly to conclusions of usefulness if the subjects of the study were closer to us (i.e. if they weren't already assumed to have done something wrong to be in that situation).
--
As for the bad outcomes, I wasn't so much talking at the system at the current instant, but more about the feedback loop this creates: suppose the rating is biased against a community of people such that their measured score is lower than their actual likelihood to show up, then the judge is less likely to give them a chance to be out, which then reinforces the social bias against the category of people and drives what was an initial bias to be a systemic issue.
My masters is in Control Theory so I tend to see systems, feedback loops, and stability of trajectories everywhere, I guess.
I think, also, that chatbots are an unfair comparison. Those are specifically designed to decrease triage time for non-emergency scenarios. This is a fundamentally different scenario, where it provides evidence to counter the prosecution. It will always fall to the judge to weigh the score versus what evidence the prosecution provides. If the prosecution has no good reason to demand punishingly high bail, and the model continues to hold up well over time, I see no reason to assume that using it leads to bad outcomes. Remember that the most overworked people are the public defenders, and this is an aide for them.