TanteTara

Are those the same people that will add security, maintainability, performance and logging "later"?

Well, just throw the constitution in the bin then (any democratic constitution, really). Might be fine for dictators though, getting rid of those unreliable judges.

I miss any discussion about the elephant in the room: On what data does a potential algorithm base its decision? Who puts in that data?

If you know how the algorithm works (and this one must be open source), you can game it by manipulating the data it gets to see (or not) and how the specific wording is.

Can the algorithm generate and ask questions by itself? Again that would be pointless if you know the algorithm - you would know the question beforehand and how exactly a given answer influences the outcome.

This idea cannot ever work until you have a strong general AI (which you can not predict any more and which may have its own biases) and we are as far from a strong general AI as we have ever been.

It isn't. Climate is a statistics game. You cannot prove or disprove that a specific dice throw ended up 6 because the dice was unfair, but you can still make robust (and thus falsifiable) predictions over hundreds of throws.