Rather than explaining how the machine learning system works as an "explanation", perhaps auditing could be added to machine learning algorithms so that you very much could produce a very long but accurate description of the process, a la pages full of "X was compared to Y, X was larger, and thus we move on to step 261". A bit like disassembling machine code.

Of course, in machine learning, the datasets backing up the comparisons could not be shared as they contain variations of confidential and personal data, but you might still end up with a legally tolerable record of the algorithmic steps involved in the decision making process even if they're not useful to see.