Regarding more verbose Pytorch code for the update step, compare:
In Tensorflow:
loss = tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=output_logits)
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss)
sess.run(optimizer)
In PyTorch:
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)
loss = nn.CrossEntropyLoss()(output, label)
optimizer.zero_grad()
loss.backward()
optimizer.step()
In my opinion, PyTorch makes the parameter update process a lot easier to understand, control, and modify (if needed). For example what if you want to modify gradients right before the weight update? In PyTorch I'd do it right here in my code after the loss.backward() statement, while in TF I'd have to modify the optimizer code. Which option would you prefer?
[1] https://stackoverflow.com/questions/44428784/when-is-a-pytor...
I’ve definitely never had to do that. Where do you get this from?
In PyTorch, you still do have to define the backward function and worry about bookkeeping the gradient, clearing gradient values at the appropriate time, and explicitly calling to calculate these things in verbose optimizer invocation code.
I encourage you to check out how this works in Keras, because it is simply just factually different than what you are saying, in ways that are specifically designed to remove certain types of boilerplate or overhead or bookkeeping that are required by PyTorch.