It eschews back-propagation but not gradient calculation. You still have to nudge the activations' weights upward for positive examples and downward for negative ones. Positive examples should give a long vector and negative ones a small vector.

Let's suppose that you are correct, which direction are the weights updated towards?

The implementations of this compute gradients locally.

It makes sense that all gradients are local. Does it make sense to say that gradient propagation through the layers is memoryless?

In my opinion, yes if and only if the update does not use a stateful optimiser, and the computation is easy / simple enough that the updated parameter value can be computed immediately.

In linear layers, it is possible. Once you have computed the gradient of the output of the vector ith vector, so a scalar, you scale the input by that value and add it to the parameters.

This is a simple FMA op: a=fma(eta*z, x, a), with z the gradient of the vector, x the input, a the parameters, and eta the learning rate. This computes a = a + eta*z*x in place.