You don't necessarily need to know the whole model, but you need accurate enough prediction score outputs for each category. I think if you have that, that's enough to numerically approximate the gradient of the output scores and perform gradient descent.