Softmax in attention is, well, a smoothed max function, so it gets you branching kinda... You're choosing which memory cell to use.