I guess I come from the AI safety side of algorithmic understanding. That is, understanding the algorithm give us zero understanding if an output is going to have a 'safe' or 'unsafe' response.

This is something important that a layman may miss. The typical line of human thought works along the lines of "understand = control" whereas in this case in this case understanding the algorithm tells us nothing about the probability that the next pull of the lever will shit out a paperclip maximizer that will begin converting the matter I covet so.