But what if, due to historical happenstance, the data is already a proxy for groups split on genetics, biology, or birth? Using that data in a way that might create a feedback loop that propagates that happenstance is entirely likely.

Part of the problem is that, when dealing with humans as the data set, the data is really messy and there's lots of noise correlated to historical circumstance.

Easier said than done.

I know, this is why all this algorithms should not be hidden/proprietary. At least we have a chance to spot why/how the computer decided my fate.

The problem is that in many cases (especially with deep learning), "the algorithm" is essentially a black box. There often isn't a straightforward way to determine exactly what the algorithm is actually learning.

I know that too, but if I(or the community) have access to it I can prove the algorithms is flawed. Say i change a bit the input and notice a huge output, like I change my non enlgish name to an english name and by magic my score triples then I prove the algorithm is discriminating me(same if I change my birth place or address).

I don't think that opening the algorithms is a sufficient condition but I think is necessary.